From Text to Embodiment: I Built a Real-time 3D Interactive Body for an AI Agent

After the last article ["AI Story King"] (https://juejin.cn/post/7661439504083582976) was published, no one asked how the large model writes stories. Instead, the most questions were about how the digital human is driven, whether it can be interrupted, and what the latency actually is.

In fact, behind these questions lies one thing: how to equip a plain-text Agent with a 3D body capable of real-time interaction. In this article, we'll discuss the integration details of the Mofa Nebula Embodied Driving SDK and a few pitfalls I encountered in the project.

1. What the Agent Lacks Isn't a Brain, but a Body

Large models give an Agent a brain to think, but the chat-box form is too barren: users can only stare at text, and the interaction is one-way. The same goes for pre-recorded videos or traditional video-stream digital humans. If we interject mid-sentence, it can only finish playing the current clip and cannot respond instantly.

To make an Agent truly chat with you like a person, it needs a complete embodied interactive intelligence system: you give it a piece of text, and it immediately and synchronously outputs speech, mouth shapes, eye movements, and body gestures, while also being interruptible at any time to re-broadcast. This multimodal interaction capability cannot be achieved by relying on large models or simple front-end code; it requires a dedicated, standardized foundation.

2. Core Technology Stack

The tech stack for AI Story King is quite ordinary, with most of the code generated with the assistance of Trae:

• Front-end Framework: Vue 3. Actually, pure static pages work too; a framework isn't strictly necessary.
• AI Coding: Trae Work, used to quickly generate code based on requirements.
• Large Model: OpenAI-compatible interfaces like DeepSeek / Tongyi Qianwen, switchable on demand.
• 3D Digital Human: Mofa Nebula Embodied Driving SDK.

The most core capability here is provided by Mofa Nebula. It is positioned as an open platform for embodied interactive intelligence, packaging the full chain of core capabilities—multimodal perception, large model agent scheduling, and LAM 3D three-layer embodied interactive intelligence—into a standardized SDK. With it, as long as the terminal supports a browser kernel, you can integrate a 3D digital human with AI interaction capabilities, without needing to train a motion model yourself.

The LAM (Large Action Model) here, as the signature underlying technology of embodied interactive intelligence, converts text into speech, mouth shapes, expressions, and motion parameters. On the front end, you just need to new XmovAvatar().

After I got the whole chain running smoothly, we only need to say one sentence, and after the large model responds, the digital human can start replying with a delay of about 500ms. The response speed is very fast.

3. SDK Integration: From Initialization to Interruptible Broadcasting

The core logic of the implementation can be divided into three steps: initializing the connection, interruptible broadcasting, and large model integration with multi-avatar switching.

3.1 Initialization and Connection

In the previous step, we already registered on the Nebula Open Platform. Next, we can create an application and obtain the appId and appSecret.

The first time the SDK connects, it needs to download avatar resources. The time taken depends on the network speed; the first time is expected to take about 10 seconds. The second time is much faster, basically connecting within 2 seconds.

The second time will be much faster, basically maintaining a connection success within 2 seconds:

I was worried about the page getting stuck during network jitter, so I added a 5-second timeout fallback:

// src/composables/useAvatarSDK.js
function createSdkInstance({ appId, appSecret }) {
  sdk.value = new window.XmovAvatar({
    containerId: "#avatar-container",
    appId,
    appSecret,
    gatewayServer: "https://nebula-agent.xingyun3d.com/user/v1/ttsa/session",
    onMessage: (msg) => {
      if (msg && msg.error_code && _sessionReject) {
        _sessionReject(msg.error_reason || "Authentication failed");
      }
    },
    onWidgetEvent: (eventName, data) => handleWidgetEvent(eventName, data),
  });
}

async function doConnect({ appId, appSecret }) {
  if (sdk.value) {
    console.warn("[SDK] Existing instance found, cleaning up first");
    destroySdk();
  }

  status.value = "connecting";
  createSdkInstance({ appId, appSecret });

  await sdk.value.init({
    onDownloadProgress: (p) => { progress.value = p; },
  });

  // Wait for gateway session confirmation, 5s timeout fallback
  await new Promise((resolve, reject) => {
    _sessionReject = reject;
    setTimeout(() => {
      if (_sessionReject) {
        _sessionReject = null;
        resolve();
      }
    }, 5000);
  });

  status.value = "connected";
  isConnected.value = true;
}

The SDK initialization has many parameters; the official documentation has a complete list. During actual integration, focus on the required fields: containerId, appId, appSecret, and gatewayServer.

For detailed parameters, refer to the Embodied Driving SDK Official Documentation.

Another special note: when not conversing, you must, must, must disconnect from the SDK, otherwise it will continuously consume credits. I previously consumed a large number of credits during debugging because I didn't disconnect for a long time.

3.2 Interruptible Broadcasting

If a child suddenly interrupts while listening to a story, the digital human needs to shut up immediately. This is a very normal logic.

The SDK has an interactiveidle() method that can switch the digital human from a broadcasting state to an interruptible interactive standby state. Sending new content afterward won't mess up the mouth shapes:

function speakWithInterrupt(text, isStart = true, isEnd = true) {
  if (!sdk.value || !isConnected.value) return;
  sdk.value.interactiveidle();
  setTimeout(() => {
    sdk.value.speak(text, isStart, isEnd);
  }, 150);
}

Different avatars may have different animation state machines. A delay of about 150ms is an empirical value I found through testing with the current few digital human avatars; it may not be universally applicable. For more complex avatars, further fine-tuning might be needed.

3.3 Large Model Integration and Multi-Avatar Switching

The large model supports OpenAI-compatible interfaces, with DeepSeek and Tongyi Qianwen switchable on demand. To support multiple cartoon avatars, I store the AppID/AppSecret and avatar presets together in useMagicKeys.js. Different model configurations can be added, edited, and switched in the settings panel:

// src/composables/useMagicKeys.js
export function useMagicKeys() {
  const configs = ref([]);
  const activeId = ref("");

  const activeConfig = computed(() =>
    configs.value.find((c) => c.id === activeId.value)
  );

  function addConfig(config) {
    const item = { id: generateId(), ...config };
    configs.value.push(item);
    activeId.value = item.id;
    persist();
  }

  // ...
}

Each configuration is not just an AppID/AppSecret, but also includes avatar presets: emoji, theme color, and empty state background. When switching related models, the UI theme and digital human avatar change together. The configuration list is stored in localStorage and won't be lost on refresh.

Actually, calling the large model is just a few lines, dynamically reading the current configuration and assembling a child-friendly system prompt:

// src/utils/chat.js
export async function chat(prompt) {
  const config = getActiveAiConfig();

  const res = await fetch(`${config.baseUrl}/v1/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${config.apiKey}`,
    },
    body: JSON.stringify({
      model: config.model,
      messages: [
        {
          role: "system",
          content: "You are a kind and lovely Story King companion, dedicated to telling stories for children...",
        },
        { role: "user", content: prompt },
      ],
    }),
  });

  const data = await res.json();
  const msg = data.choices?.[0]?.message;
  let reply = msg?.content || "";
  return reply;
}

The model configuration is written as follows:

// src/constants/aiModels.js
export const AI_PROVIDERS = [
  {
    id: "qwen",
    name: "Tongyi Qianwen (Alibaba Cloud Bailian)",
    defaultModel: "qwen-plus",
    baseUrl: "https://dashscope.aliyuncs.com/compatible-mode",
  },
  {
    id: "deepseek",
    name: "DeepSeek",
    defaultModel: "deepseek-chat",
    baseUrl: "https://api.deepseek.com",
  },
];

4. Parameter Stream + Client-Side Rendering Comparison

First, an imprecise analogy to help you understand instantly:

• Video Stream = Watching a live stream. You only see the picture after the streamer finishes speaking. High latency, bandwidth-hungry, and cannot be interrupted.
• Parameter Stream = Playing a game. The character model is on your local machine; the server only syncs action commands. Lightweight and flexible.

In a local Chrome browser + Wi-Fi environment, the end-to-end latency from when I finish speaking to when the digital human starts talking is stable at around 500ms. The core reason the latency can be pushed this low is this combination punch of parameter stream + client-side rendering.

4.1 Comparison of Two Solutions: See the Difference at a Glance

Dimension	Video Stream Digital Human	Parameter Stream Digital Human
Bandwidth Consumption	High	Low (audio + parameter packets only)
End-to-End Latency	Typically > 1000ms	Stable around 500ms
Interruptibility	Poor (must wait for current clip to finish)	Strong (interruptible and replayable at any time)
Server Cost	High (one GPU render path per user)	Low (only speech synthesis + parameter generation)
Cross-End Adaptation	Difficult (rendering pipeline must be rewritten for different terminals)	Easy (one set of call logic covers all terminals)

4.2 500ms Latency Breakdown: Where Does the Time Go?

The latency wasn't magically reduced. Based on my actual test estimates, it can be roughly broken down into three segments:

1. Speech Synthesis: The server converts text into audio parameters. This is the bulk of it.
2. Parameter Delivery: Lightweight data like mouth angles, head turns, and gesture numbers are transmitted.
3. Client-Side Rendering: Local WebGL drives the 3D model to open its mouth, blink, and move its arms based on the parameters.

The key to the parameter stream is that the 3D avatar model and motion library are downloaded locally during initialization. Subsequent communication no longer needs to transmit full images, only numerical commands like "open mouth 60°" or "turn head left 15°".

5. A Few Pitfalls I Encountered

During the integration process, I stepped on a few specific pitfalls, which I'm sharing here to help you avoid them.

5.1 Not Cleaning Up the Instance Before Reconnecting Causes Double Rendering Layers

After switching the AppID, two digital humans appeared on the page, stacked on top of each other, with severe screen flickering. The reason is that the SDK instance internally binds a DOM container and a WebGL context. If you directly new a new instance without calling destroy(), the old rendering loop won't stop.

This pitfall sounds simple, but it's easy to miss in state management. I clean up the old instance at the beginning of doConnect and also added destroySdk() in onBeforeUnmount to prevent residual rendering threads after the component is destroyed:

function destroySdk() {
  if (sdk.value) {
    sdk.value.destroy();
    sdk.value = null;
    isConnected.value = false;
    status.value = "idle";
  }
}

5.2 No Retry Mechanism for Resource Loading Failure on Weak Networks

When loading avatar resources for the first time, if a network fluctuation causes the download to fail, the SDK does not automatically retry, and the page will remain stuck in a loading state. I added a retry logic to maximize the chance of a successful first connection on weak networks:

async function doConnect({ appId, appSecret }, retryCount = 0) {
  try {
    // ... connection logic
  } catch (err) {
    if (retryCount < 3) {
      console.warn(`[SDK] Connection failed, retry ${retryCount + 1}`);
      await new Promise(r => setTimeout(r, 2000));
      return doConnect({ appId, appSecret }, retryCount + 1);
    }
    throw err;
  }
}

Conclusion

Large models enable AI to think, but to truly bring it into screens, robots, exhibition halls, and educational scenarios, a complete embodied interactive intelligence is needed to realize a visual, real-time communicable AI carrier. The Mofa Nebula Embodied Driving SDK is the key tool to complete this layer.

The project effect and complete build ideas from the last article are here: ["AI Story King"] (https://juejin.cn/post/7661439504083582976)

To experience the online demo, visit https://aistory.uviewpro.cn/