跪拜 Guibai
← Back to the summary

A 500ms-Latency 3D Storyteller That Kids Can Interrupt

A while ago, my kid wanted to hear stories before bed. I started by using ChatGPT to generate text and then played it with a text-to-speech voice, but he ran off after a few sentences. The reason is simple: plain text and pre-recorded videos lack embodied interactive intelligence. They can only output one-way; they can't achieve real-time, two-way empathetic interaction between a human and an AI. Many digital human solutions on the market have high latency, can't be interrupted, and are expensive, making them completely unsuitable for this kind of real-time interactive scenario.

So I decided to build my own "Story King Companion": Vue 3 + Vite for the frontend, Trae Work for AI Coding to generate the skeleton, DeepSeek/Tongyi Qianwen as the brain for story reasoning and creation, and Mofa Nebula to provide complete embodied interactive intelligence. To clarify the division of labor in one sentence: the large language model teaches the AI to think, and Mofa Nebula gives the AI a perceptible, interactive 3D body with real-time human-like expression capabilities. When the kid says, "I want to hear a story about a little rabbit," the large model generates the story, and the 3D cartoon digital human broadcasts it in real-time with synchronized lip movements, expressions, and actions. It also supports voice input, switching avatars, and use on mobile phones.

The end-to-end latency of the entire chain is about 500ms. There is only one core codebase. Once the on-screen digital human is running, it can be reused for humanoid service robots.

0.gif

1. Project Effect and Overall Architecture

The final effect is this: open the webpage, the left side is a 3D digital human stage, and the right side is a chat panel. Kids can type or press and hold the microphone to speak. After they finish, the large model generates a Chinese story, and the digital human immediately starts telling it, with its mouth and expressions moving along with the content. After finishing, it can continue to answer follow-up questions.

I broke down the main capabilities into several parts:

2.png

The entire call chain is very short: User Input -> Speech Recognition to Text -> chat.js calls the large model -> Gets the story text -> speakWithInterrupt sends it to the Nebula SDK -> The digital human starts broadcasting. The key time-consuming parts of the chain are only the large model generation time and the SDK response time. Nebula uses a parameter stream + client-side rendering, which can reduce the latency to about 500ms.

3.png

2. Building the Skeleton with Trae Work

This project wasn't built line-by-line from scratch; it was completely implemented from zero using Trae Work. I'm sure everyone has used Trae IDE by now, and Trae Work is an AI-native workbench for a broader audience, available on web/desktop/mobile, supporting three modes: Work, Code, and Design.

18.png

15.png

I first used Trae Work to clearly describe the requirements: "A Vue 3 3D digital human story companion page, with a stage on the left and chat on the right, supporting voice input and multi-model switching." Trae Work helped me generate the basic directory, component structure, and composable prototypes, which I then integrated and tuned according to the Mofa Nebula SDK documentation.

The core files generated by Trae Work are roughly as follows:

The place I used Trae Work the most was for generating boilerplate code and edge-case handling. For example, the voice input composable needs to handle scenarios like unsupported browsers, denied permissions, no speech detected, and network errors. It's easy to miss these when writing by hand, but Trae Work can provide a relatively complete structure in one go, which I then fine-tuned based on business logic.

// src/composables/useSpeechRecognition.js
export function useSpeechRecognition(options = {}) {
  const SpeechRecognition =
    window.SpeechRecognition || window.webkitSpeechRecognition;

  const isSupported = ref(!!SpeechRecognition);
  const isListening = ref(false);
  const transcript = ref("");
  const error = ref(null);

  function start() {
    // Create a new instance each time to avoid state confusion
    if (recognition) {
      try { recognition.abort(); } catch (e) {}
    }
    recognition = createRecognition();
    transcript.value = "";
    recognition.start();
  }

  function stop() {
    if (recognition && isListening.value) {
      try { recognition.stop(); } catch (e) {}
    }
  }

  // ... onresult / onerror / onend handling
}

5.png

There's a detail here: the Web Speech API works well in Chrome, but compatibility in Safari and on mobile is uneven, so I added an isSupported check. When it's not supported, the button automatically hides to prevent users from clicking it and getting no response.

3. Large Model Integration: More Than Just Calling an API

For the story's brain, I chose DeepSeek and Tongyi Qianwen as the main engines. In the project, I made multiple models configurable to avoid having to change code if a model becomes unstable or its price increases later.

The model configuration is written in src/constants/aiModels.js, and all interfaces are in an OpenAI-compatible format:

// src/constants/aiModels.js
export const AI_PROVIDERS = [
  {
    id: "qwen",
    name: "Tongyi Qianwen (Alibaba Cloud Bailian)",
    defaultModel: "qwen-plus",
    proxyPrefix: "/proxy-api/qwen",
    target: "https://dashscope.aliyuncs.com/compatible-mode",
  },
  {
    id: "deepseek",
    name: "DeepSeek",
    defaultModel: "deepseek-chat",
    proxyPrefix: "/proxy-api/deepseek",
    target: "https://api.deepseek.com",
  },
  // ...
];

During local development, Vite's proxy forwards these prefixes to the corresponding official addresses, preventing the frontend from directly exposing the real API domain:

// vite.config.js
server: {
  proxy: {
    "/proxy-api/deepseek": {
      target: "https://api.deepseek.com",
      changeOrigin: true,
      rewrite: (path) => path.replace(/^\/proxy-api\/deepseek/, ""),
    },
  },
}

Runtime configuration prioritizes reading from localStorage, falling back to .env if not found:

// src/utils/aiConfig.js
export function getActiveAiConfig() {
  const saved = localStorage.getItem(STORAGE_KEY);
  if (saved) {
    const parsed = JSON.parse(saved);
    const provider = findProvider(parsed.providerId);
    if (provider && parsed.apiKey) {
      return {
        providerId: provider.id,
        apiKey: parsed.apiKey,
        model: parsed.model || provider.defaultModel,
        proxyPrefix: provider.proxyPrefix,
      };
    }
  }

  return {
    providerId: "deepseek",
    apiKey: import.meta.env.VITE_AI_API_KEY || "",
    model: import.meta.env.VITE_AI_MODEL || "deepseek-chat",
    proxyPrefix: "/proxy-api/deepseek",
  };
}

The place where the large model is called, chat.js, is very simple. It dynamically reads the current configuration and assembles a system prompt to make the model tell stories in a child-friendly tone:

// src/utils/chat.js
export async function chat(prompt) {
  const config = getActiveAiConfig();

  const res = await fetch(`${config.proxyPrefix}/v1/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${config.apiKey}`,
    },
    body: JSON.stringify({
      model: config.model,
      messages: [
        {
          role: "system",
          content:
            "You are a kind and lovely Story King Companion, specializing in telling stories for children...",
        },
        { role: "user", content: prompt },
      ],
    }),
  });

  // ...
}

I hit a pitfall here: a certain model of Tongyi Qianwen returned an empty content field, with the actual story content hidden in the reasoning field. Moreover, the reasoning often had an English thought process at the beginning. I had to write an extractStoryFromReasoning function to extract the Chinese text based on several patterns, such as matching the *(Story Body)* marker, matching Chinese characters after "Drafting the Story", or directly finding the first continuous Chinese paragraph.

function extractStoryFromReasoning(reasoning) {
  const sectionMatch = reasoning.match(
    /(?:\*\(Greeting\)\*|\*\(Story Body\)\*)\s*([\s\S]*?)(?=\n\s*\*\(|$)/
  );
  if (sectionMatch) return sectionMatch[1].trim();

  const draftingMatch = reasoning.match(
    /(?:Drafting the Story)[\s\S]*?\n\s*([\u4e00-\u9fa5][\s\S]*?)(?=\n\s*\*\*Check|$)/
  );
  if (draftingMatch) return draftingMatch[1].trim();

  const chineseBlockMatch = reasoning.match(/[\u4e00-\u9fa5]{10,}[\s\S]*$/);
  if (chineseBlockMatch) return chineseBlockMatch[0].trim();

  return reasoning;
}

This pitfall cost me half a day, because initially I thought I had written the prompt wrong. It was only after opening the browser console and seeing the raw response that I found the problem.

6.png

4. Mofa Nebula Embodied Driving SDK: "Performing" the Story

A large model can only output text. To make a digital human "tell" a story, a layer of "body" is needed. I integrated the Mofa Nebula XmovAvatar SDK, which is the Mofa Nebula Embodied Driving SDK.

10.png

Mofa Nebula is not just a simple 3D avatar generation tool, but an open underlying platform for full-domain embodied interaction in the AI terminal era. Its core value is providing standardized embodied interactive intelligence for all large models and Agents to land. The platform is built on a self-developed LAM text-to-3D multimodal large model as its base, constructing a three-layer closed-loop architecture of "Multimodal Perception Layer - Large Model Agent Layer - LAM3D Embodied Expression Layer." Relying on a self-developed parameter stream architecture + AI client-side rendering and solving, it doesn't need pre-recorded videos. Inputting text can drive a 3D avatar in real-time with synchronized voice, eye gaze, micro-expressions, and full-body movements, truly realizing complete embodied interactive intelligence that can perceive, provide feedback, and be interrupted in real-time.

9.png

The integration process roughly involves three steps: register an application on the Mofa Nebula open platform to get an AppID and AppSecret; create an XmovAvatar instance on the frontend and complete the init() initialization, waiting for resource downloads and gateway session confirmation; then call the speak() method to make the digital human speak.

Below are a few key code snippets from my project:

SDK initialization is wrapped in useAvatarSDK.js:

// src/composables/useAvatarSDK.js
function createSdkInstance({ appId, appSecret }) {
  sdk.value = new window.XmovAvatar({
    containerId: "#avatar-container",
    appId,
    appSecret,
    gatewayServer: "https://nebula-agent.xingyun3d.com/user/v1/ttsa/session",
    onMessage: (msg) => {
      if (msg && msg.error_code && _sessionReject) {
        _sessionReject(msg.error_reason || "Authentication failed");
      }
    },
    onWidgetEvent: (eventName, data) => {
      handleWidgetEvent(eventName, data);
    },
  });
}

When connecting, first init to download resources, then wait for gateway session confirmation, and finally set the status to connected:

async function doConnect({ appId, appSecret }) {
  if (sdk.value) {
    console.warn("[SDK] Existing instance, cleaning up first");
    destroySdk();
  }

  status.value = "connecting";
  createSdkInstance({ appId, appSecret });

  await sdk.value.init({
    onDownloadProgress: (p) => {
      progress.value = p;
    },
  });

  // Wait for gateway session confirmation, 5s timeout fallback
  await new Promise((resolve, reject) => {
    _sessionReject = reject;
    setTimeout(() => {
      if (_sessionReject) {
        _sessionReject = null;
        resolve();
      }
    }, 5000);
  });

  status.value = "connected";
  isConnected.value = true;
}

A small pitfall here: if you connect again without cleaning up the old instance after a successful connection, the SDK will throw an error or create two rendering layers. So I added a check at the beginning of doConnect to destroySdk() if an old instance exists.

When broadcasting a story, to allow the user to interrupt at any time, I use speakWithInterrupt:

function speakWithInterrupt(text, isStart = true, isEnd = true) {
  if (!sdk.value || !isConnected.value) return;
  sdk.value.interactiveidle();
  setTimeout(() => {
    sdk.value.speak(text, isStart, isEnd);
  }, 150);
}

interactiveidle puts the digital human into an interruptible state first, then speaks the new content. Instant interruption is a key capability of embodied interactive intelligence. This way, if a child suddenly asks a new question while listening to a story, the digital human will immediately stop and answer, instead of finishing the old story.

To make the interface less monotonous, I also implemented multi-avatar switching: a little monkey, a little bear, a little rabbit, etc. Each avatar has a corresponding emoji, theme color, and empty-state gradient background. The configuration is stored in useMagicKeys.js, and parents can add, edit, and switch different keychains in the settings panel.

14.png

// src/composables/useMagicKeys.js
export function useMagicKeys() {
  const configs = ref([]);
  const activeId = ref("");

  const activeConfig = computed(() =>
    configs.value.find((c) => c.id === activeId.value)
  );

  function addConfig(config) {
    const item = { id: generateId(), ...config };
    configs.value.push(item);
    activeId.value = item.id;
    persist();
  }

  // ...
}

5. Experience the Deployed Story King Application

If you don't want to run the code locally for now, you can directly open the online version I've deployed:

https://aistory.uviewpro.cn/

After opening, the left side is the 3D digital human stage, and the right side is the chat panel. On the first entry, the digital human will show an empty state of "still sleeping," requiring parents to help complete two initialization steps first.

Usage Steps

  1. Connect the Mofa Nebula Magic Key

    Click the settings entry on the page (top right on desktop, top of the card on mobile) to enter "Mom and Dad's Magic Keychain." Fill in the AppID and AppSecret you applied for on the Mofa Nebula open platform, then click "Summon the Story King." Wait a few seconds for the resource download and gateway session confirmation to complete, and the digital human will wake up from its sleeping state and appear on the stage.

17.png

  1. Set the Large Model API Key

    Also in the settings panel, choose your preferred large model, such as DeepSeek or Tongyi Qianwen, and fill in the corresponding API Key. This Key is only saved in the browser's local localStorage and will not be transmitted to my server. Once configured, the digital human will have a "brain."

16.png

  1. Start Chatting or Random Story

    Now the child can start talking. Click the microphone button next to the input box, press and hold to speak, and the text will automatically fill into the input box upon release; you can also type directly. Say something like "I want to hear a story about a little rabbit," the large model generates the story, and the digital human immediately starts broadcasting. If unsure what to listen to, there's also a "Story Dice" button in the panel; click it for a surprise.

  2. Switch Avatars and Adjust Volume (Optional)

    In the settings panel, you can switch between different avatars like a little monkey, little bear, or little rabbit, each with its own theme color. The volume of the digital human can also be independently adjusted at the top of the chat panel, avoiding mixing with the system volume.

It is recommended to experience it in the Chrome browser, where the Web Speech API's recognition effect is most stable. If the voice recognition button doesn't appear in Safari or on mobile, it means the current environment doesn't support it, and you can also interact by typing directly.

Mofa Nebula new users can register with the invitation code JMFASQDPXB to get 1000 credits, enough for a long period of trial.

Child-Friendly Design

Since it's for children, technical terms must be hidden. I centralized all child-facing text into src/constants/kidsText.js. For example, AppID is called "Magic Key Number," AppSecret is called "Magic Spell," and disconnecting is called "Let the King Rest."

// src/constants/kidsText.js
export const KIDS_TEXT = {
  CONFIG_TITLE: "🔑 Mom and Dad's Magic Keychain",
  BTN_CONNECT: "✨ Summon the Story King",
  BTN_DISCONNECT: "💤 Let the King Rest",
  KEY_APP_ID_LABEL: "Magic Key Number",
  KEY_APP_SECRET_LABEL: "Magic Spell",
};

In terms of interaction, I also tried to align with children's habits and support responsive layout:

PC UI:

11.png

12.png

13.png

Mobile UI:

7.png

8.png

6. Extension

Only after truly getting it running did I realize that the key to whether a digital human attracts children isn't how smart the large model is, but whether it can respond instantly like a real person. Children won't patiently wait for a chat box to slowly type out words; they want a "face-to-face" interactive feeling.

The current AI industry is completing a transformation from pure text dialogue to full-domain embodied interactive intelligence. AI that relies only on a chat box finds it hard to enter real offline scenarios like education, stores, and robots. Mofa Nebula provides standardized underlying capabilities, allowing any developer to equip a large model with a 3D body possessing complete embodied interactive intelligence at a low cost.

I've only made a web version for now, but this experience makes me feel that in the future, if the same interaction logic is put into service robots, AR glasses, or even humanoid robots at home, it might open up many more interesting scenarios. For me personally, the Story King is more of a starting point: it let me see that a large model isn't just a tool for answering questions, but can also become a visible, interactive, emotional little character.

Next, I also want to try to see if the digital human can recognize the child's vocal emotions, or automatically switch expressions and actions based on the conversation content. These capabilities don't seem so far away now.

7. Summary

After completing the entire project, my biggest feeling is: Trae Work is responsible for quickly setting up the engineering skeleton, lowering the barrier to front-end engineering development. The large model is responsible for generating content, giving the AI the ability to think and create. The Mofa Nebula SDK is responsible for "performing" the content, providing a complete set of standardized underlying capabilities for embodied interactive intelligence. The division of labor among the three is very clear. Without Trae Work, I might still be struggling with directory structure and component division; without the Nebula SDK, the Story King would just be a text chat robot, much less attractive to children.

If you want to try it too, you can use my invitation code JMFASQDPXB to register on the Mofa Nebula Open Platform. New users can get 1000 credits, enough for many rounds of testing.

Finally, I'd like to discuss a question with everyone:

In what scenarios would you use a 3D digital human? Educational companionship, customer service guidance, or humanoid robots?

Feel free to share in the comments.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

前端梦工厂

An LLM writes the story, an AI 3D digital human performs it in real time — the experience is pretty good.