跪拜 Guibai
← Back to the summary

A 500ms Pipeline Turns LLMs Into Real-Time 3D Storytellers That Kids Can Interrupt

A while ago, my kid wanted to hear stories before bed. I started by generating text with ChatGPT and reading it aloud with TTS, but he ran off after a few sentences. The reason is simple: plain text and pre-recorded videos lack embodied interactive intelligence; they can only output one-way, unable to achieve real-time, two-way empathetic interaction between human and AI. Many digital human solutions on the market suffer from high latency, no interruption capability, and high costs, making them completely unsuitable for this kind of real-time interactive scenario.

So I decided to build a "Story King Companion" myself: Vue 3 + Vite for the frontend, Trae Work for AI Coding to generate the skeleton, DeepSeek/Tongyi Qianwen as the brain for story reasoning and creation, and Mofa Nebula providing the complete embodied interactive intelligence. To clarify the division of labor in one sentence: the large model enables AI to think, and Mofa Nebula gives AI a perceivable, interactive 3D body with real-time anthropomorphic expression capabilities. The kid says, "I want to hear a story about a bunny," the large model generates the story, and the 3D cartoon digital human broadcasts it in real-time with lip-sync, expressions, and actions, while also supporting voice input, avatar switching, and mobile use.

The end-to-end latency of the entire chain is around 500ms. There's only one core codebase; once the on-screen digital human works, it can be reused for humanoid service robots.

0.gif

1. Project Effect and Overall Architecture

The final effect is this: open the webpage, the left side is the 3D digital human stage, and the right side is the chat panel. Kids can type or hold down the microphone to speak; after speaking, the large model generates a Chinese story, and the digital human immediately starts telling it, with mouth and expressions moving along with the content. After finishing, they can continue asking follow-up questions.

I broke down the main capabilities into several parts:

2.png

The entire call chain is very short: User Input -> Speech Recognition to Text -> chat.js calls the large model -> Get story text -> speakWithInterrupt passes it to the Nebula SDK -> Digital human starts broadcasting. The key time-consuming parts in the chain are only the large model generation time and the SDK response time. Nebula uses parameter streaming + client-side rendering to push latency down to about 500ms.

3.png

2. Building the Skeleton with Trae Work

This project wasn't built line-by-line from scratch; it was entirely implemented from scratch using Trae Work. You've probably all used Trae IDE by now, and Trae Work is an AI-native workbench for a broader audience, offering Web/Desktop/Mobile versions and supporting Work, Code, and Design modes.

18.png

15.png

I first used Trae Work to clearly describe the requirements: "A Vue 3 3D digital human story companion page, with a stage on the left and chat on the right, supporting voice input and multi-model switching." Trae Work helped me generate the base directory, component structure, and composable prototypes, which I then adapted and tuned based on the Mofa Nebula SDK documentation.

The core files generated by Trae Work are roughly as follows:

I used Trae Work most for generating boilerplate code and edge-case handling. For example, the voice input composable needs to handle scenarios like browser not supported, permission denied, no speech detected, network errors, etc. It's easy to miss things when writing it manually; Trae Work can provide a relatively complete structure in one go, and I just fine-tune it based on business needs.

// src/composables/useSpeechRecognition.js
export function useSpeechRecognition(options = {}) {
  const SpeechRecognition =
    window.SpeechRecognition || window.webkitSpeechRecognition;

  const isSupported = ref(!!SpeechRecognition);
  const isListening = ref(false);
  const transcript = ref("");
  const error = ref(null);

  function start() {
    // Create a new instance each time to avoid state confusion
    if (recognition) {
      try { recognition.abort(); } catch (e) {}
    }
    recognition = createRecognition();
    transcript.value = "";
    recognition.start();
  }

  function stop() {
    if (recognition && isListening.value) {
      try { recognition.stop(); } catch (e) {}
    }
  }

  // ... onresult / onerror / onend handling
}

5.png

One detail here: the Web Speech API works well in Chrome, but compatibility in Safari and on mobile is uneven, so I added an isSupported check. When unsupported, the button automatically hides to prevent users from clicking it and getting no response.

3. Large Model Integration: More Than Just Calling an API

For the story's brain, I chose DeepSeek and Tongyi Qianwen as the main forces. In the project, I made multiple models configurable to avoid having to change code if a model becomes unstable or its price increases later.

Model configuration is written in src/constants/aiModels.js, all interfaces are in OpenAI-compatible format:

// src/constants/aiModels.js
export const AI_PROVIDERS = [
  {
    id: "qwen",
    name: "Tongyi Qianwen (Alibaba Cloud Bailian)",
    defaultModel: "qwen-plus",
    proxyPrefix: "/proxy-api/qwen",
    target: "https://dashscope.aliyuncs.com/compatible-mode",
  },
  {
    id: "deepseek",
    name: "DeepSeek",
    defaultModel: "deepseek-chat",
    proxyPrefix: "/proxy-api/deepseek",
    target: "https://api.deepseek.com",
  },
  // ...
];

During local development, Vite's proxy forwards these prefixes to the corresponding official addresses, avoiding exposing the real API domain directly from the frontend:

// vite.config.js
server: {
  proxy: {
    "/proxy-api/deepseek": {
      target: "https://api.deepseek.com",
      changeOrigin: true,
      rewrite: (path) => path.replace(/^\/proxy-api\/deepseek/, ""),
    },
  },
}

Runtime configuration prioritizes reading from localStorage, falling back to .env if not found:

// src/utils/aiConfig.js
export function getActiveAiConfig() {
  const saved = localStorage.getItem(STORAGE_KEY);
  if (saved) {
    const parsed = JSON.parse(saved);
    const provider = findProvider(parsed.providerId);
    if (provider && parsed.apiKey) {
      return {
        providerId: provider.id,
        apiKey: parsed.apiKey,
        model: parsed.model || provider.defaultModel,
        proxyPrefix: provider.proxyPrefix,
      };
    }
  }

  return {
    providerId: "deepseek",
    apiKey: import.meta.env.VITE_AI_API_KEY || "",
    model: import.meta.env.VITE_AI_MODEL || "deepseek-chat",
    proxyPrefix: "/proxy-api/deepseek",
  };
}

The place where the large model is called, chat.js, is very simple. It dynamically reads the current configuration and assembles a system prompt to make the model tell stories in a child-friendly tone:

// src/utils/chat.js
export async function chat(prompt) {
  const config = getActiveAiConfig();

  const res = await fetch(`${config.proxyPrefix}/v1/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${config.apiKey}`,
    },
    body: JSON.stringify({
      model: config.model,
      messages: [
        {
          role: "system",
          content:
            "You are a kind and lovely Story King Companion, specialized in telling stories for children...",
        },
        { role: "user", content: prompt },
      ],
    }),
  });

  // ...
}

I hit a pitfall here: a certain model of Tongyi Qianwen returned an empty content field; the actual story content was hidden in the reasoning field, and the reasoning often had an English thought process at the beginning. I had to write an extractStoryFromReasoning function to extract the Chinese text body based on several patterns, such as matching *(Story Body)* markers, matching Chinese characters after "Drafting the Story", or directly finding the first continuous Chinese paragraph.

function extractStoryFromReasoning(reasoning) {
  const sectionMatch = reasoning.match(
    /(?:\*\(Greeting\)\*|\*\(Story Body\)\*)\s*([\s\S]*?)(?=\n\s*\*\(|$)/
  );
  if (sectionMatch) return sectionMatch[1].trim();

  const draftingMatch = reasoning.match(
    /(?:Drafting the Story)[\s\S]*?\n\s*([\u4e00-\u9fa5][\s\S]*?)(?=\n\s*\*\*Check|$)/
  );
  if (draftingMatch) return draftingMatch[1].trim();

  const chineseBlockMatch = reasoning.match(/[\u4e00-\u9fa5]{10,}[\s\S]*$/);
  if (chineseBlockMatch) return chineseBlockMatch[0].trim();

  return reasoning;
}

This pitfall cost me half a day because initially I thought I had written the prompt wrong. I only found the problem after opening the browser console and seeing the raw response.

6.png

4. Mofa Nebula Embodied Driving SDK: "Performing" the Story

A large model can only output text. To make the digital human "tell" the story, a layer of "body" is needed. I integrated the Mofa Nebula XmovAvatar SDK, which is the Mofa Nebula Embodied Driving SDK.

Mofa Nebula is not just a 3D avatar generation tool, but a universal embodied interaction underlying open platform for the AI terminal era. Its core value is providing standardized embodied interactive intelligence for all large models and Agents to land.

9.png

The integration process roughly involves three steps: register an application on the Mofa Nebula Open Platform to get an AppID and AppSecret; create an XmovAvatar instance on the frontend and complete init() initialization, waiting for resource downloads and gateway session confirmation; then call the speak() method to make the digital human speak.

Tip: If you fill in the invitation code JMFASQDPXB during registration, you can get 1000 call credits.

Below are a few key code snippets from my project:

SDK initialization is wrapped in useAvatarSDK.js:

// src/composables/useAvatarSDK.js
function createSdkInstance({ appId, appSecret }) {
  sdk.value = new window.XmovAvatar({
    containerId: "#avatar-container",
    appId,
    appSecret,
    gatewayServer: "https://nebula-agent.xingyun3d.com/user/v1/ttsa/session",
    onMessage: (msg) => {
      if (msg && msg.error_code && _sessionReject) {
        _sessionReject(msg.error_reason || "Authentication failed");
      }
    },
    onWidgetEvent: (eventName, data) => {
      handleWidgetEvent(eventName, data);
    },
  });
}

When connecting, first init to download resources, then wait for gateway session confirmation, and finally set the status to connected:

async function doConnect({ appId, appSecret }) {
  if (sdk.value) {
    console.warn("[SDK] Existing instance found, cleaning up first");
    destroySdk();
  }

  status.value = "connecting";
  createSdkInstance({ appId, appSecret });

  await sdk.value.init({
    onDownloadProgress: (p) => {
      progress.value = p;
    },
  });

  // Wait for gateway session confirmation, 5s timeout fallback
  await new Promise((resolve, reject) => {
    _sessionReject = reject;
    setTimeout(() => {
      if (_sessionReject) {
        _sessionReject = null;
        resolve();
      }
    }, 5000);
  });

  status.value = "connected";
  isConnected.value = true;
}

A small pitfall here: if you connect again without cleaning up the old instance after a successful connection, the SDK will throw an error or create two rendering layers. So I added a check at the beginning of doConnect to call destroySdk() if an old instance exists.

When broadcasting a story, to allow the user to interrupt at any time, I use speakWithInterrupt:

function speakWithInterrupt(text, isStart = true, isEnd = true) {
  if (!sdk.value || !isConnected.value) return;
  sdk.value.interactiveidle();
  setTimeout(() => {
    sdk.value.speak(text, isStart, isEnd);
  }, 150);
}

interactiveidle puts the digital human into an interruptible state first, then speaks the new content, enabling instant interruption. This is a key capability of embodied interactive intelligence. This way, if a child suddenly asks a new question while listening to a story, the digital human will immediately stop and answer, instead of finishing the old story.

To make the interface less monotonous, I also implemented multi-avatar switching: little monkey, little bear, little bunny, etc. Each avatar has a corresponding emoji, theme color, and empty-state gradient background. The configuration is stored in useMagicKeys.js, allowing parents to add, edit, and switch different keychains in the settings panel.

14.png

// src/composables/useMagicKeys.js
export function useMagicKeys() {
  const configs = ref([]);
  const activeId = ref("");

  const activeConfig = computed(() =>
    configs.value.find((c) => c.id === activeId.value)
  );

  function addConfig(config) {
    const item = { id: generateId(), ...config };
    configs.value.push(item);
    activeId.value = item.id;
    persist();
  }

  // ...
}

5. Experience the Deployed Story King Application

If you don't want to run the code locally for now, you can directly open the online version I've already deployed:

https://aistory.uviewpro.cn/

After opening, the left side is the 3D digital human stage, and the right side is the chat panel. On the first visit, the digital human will show a "still sleeping" empty state, requiring parents to complete two initialization steps first.

Usage Steps

  1. Connect Mofa Nebula Magic Keychain

    Click the settings entry on the page (top right on desktop, top of the card on mobile) to enter "Mom and Dad's Magic Keychain". Fill in the AppID and AppSecret you applied for on the Mofa Nebula Open Platform, then click "Summon Story King". Wait a few seconds for resource downloads and gateway session confirmation to complete, and the digital human will wake up from sleep and appear on the stage.

  2. Set Large Model API Key

    Also in the settings panel, choose your preferred large model, such as DeepSeek or Tongyi Qianwen, and fill in the corresponding API Key. This Key is only saved in the browser's local localStorage and will not be transmitted to my server. Once configured, the digital human will have a "brain".

16.png

  1. Start Chatting or Random Story

    Now the child can start talking. Click the microphone button next to the input box, hold to speak, and release to automatically fill the text into the input box; you can also type directly. Say "I want to hear a story about a bunny", the large model generates the story, and the digital human immediately starts broadcasting. If you're not sure what to listen to, there's also a "Story Dice" button in the panel; click it for a surprise.

  2. Switch Avatars and Adjust Volume (Optional)

    In the settings panel, you can switch between different avatars like little monkey, little bear, little bunny, etc., each with its own theme color. The top of the chat panel also allows independent volume adjustment for the digital human, avoiding mixing with the system volume.

It is recommended to experience it in the Chrome browser, where the Web Speech API recognition is most stable. If the voice recognition button doesn't appear in Safari or on mobile, it means the current environment doesn't support it; you can still interact by typing directly.

Child-Friendly Design

Since it's for children, technical terms must be hidden. I centralized all child-facing text in src/constants/kidsText.js, for example, AppID is called "Magic Key Number", AppSecret is called "Magic Spell", and disconnect is called "Let the King Rest".

// src/constants/kidsText.js
export const KIDS_TEXT = {
  CONFIG_TITLE: "🔑 Mom and Dad's Magic Keychain",
  BTN_CONNECT: "✨ Summon Story King",
  BTN_DISCONNECT: "💤 Let the King Rest",
  KEY_APP_ID_LABEL: "Magic Key Number",
  KEY_APP_SECRET_LABEL: "Magic Spell",
};

Interaction-wise, I also tried to align with children's habits and support responsive layout:

PC UI:

11.png

12.png

13.png

Mobile UI:

7.png

8.png

6. Extension

Only after truly running it did I realize that whether a digital human attracts children doesn't depend on how smart the large model is, but on whether it can respond instantly like a real person. Children won't patiently wait for a chatbox to slowly type; they want a "face-to-face" interactive feeling.

The current AI industry is undergoing a transformation from pure text dialogue to universal embodied interactive intelligence. AI relying only on a chatbox struggles to enter real offline scenarios like education, retail stores, and robots. Mofa Nebula provides standardized underlying capabilities, allowing any developer to equip a large model with a 3D body possessing complete embodied interactive intelligence at low cost.

I've only made a web version so far, but this experience makes me think that in the future, if the same interaction logic is put into service robots, AR glasses, or even home humanoid robots, it might open up many more interesting scenarios. For me personally, Story King is more like a starting point: it showed me that a large model is not just a tool for answering questions, but can also become a little character that can be seen, interacted with, and has emotions.

Next, I want to try whether the digital human can recognize a child's vocal emotions, or automatically switch expressions and actions based on the conversation content. These capabilities don't seem so far away now.

7. Summary

My biggest takeaway from this entire project is: Trae Work is responsible for quickly setting up the engineering skeleton, lowering the barrier to frontend engineering development. The large model is responsible for generating content, giving AI the ability to think and create. The Mofa Nebula SDK is responsible for "performing" the content, providing a complete set of standardized underlying embodied interactive intelligence capabilities. The division of labor among the three is very clear. Without Trae Work, I might still be struggling with directory structure and component division; without the Nebula SDK, Story King would just be a text chatbot, much less attractive to children.

Finally, I want to discuss a question with everyone:

In what scenarios would you use a 3D digital human? Educational companionship, customer service guidance, or humanoid robots?

Feel free to share in the comments.