Frontend · Artificial Intelligence · AIGC

Giving a Text Agent a Real-Time 3D Body with an Embodied Driving SDK

By 前端梦工厂 · Jul 20, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

The jump from text-only chatbots to interruptible, real-time 3D agents is gated by infrastructure, not model capability. A parameter-stream SDK that offloads rendering to the client sidesteps the bandwidth and server-cost problems of video-streamed digital humans, making embodied agents practical for web apps.

Summary

A children's storytelling app gets a 3D digital human by wiring a large language model to the Mofa Nebula Embodied Driving SDK. Instead of streaming video, the SDK sends lightweight action parameters—mouth angles, head turns, gesture IDs—to a locally rendered WebGL model. The result is an avatar that can be interrupted at any time and re-broadcast new text without visual glitches. End-to-end latency from user speech to avatar response sits around 500ms, broken down into server-side speech synthesis, parameter delivery, and client-side rendering. The integration uses Vue 3 with OpenAI-compatible model backends like DeepSeek and Tongyi Qianwen, switchable through a settings panel that also swaps avatar presets and UI themes. Key implementation details include a 5-second connection timeout, a 150ms delay after calling `interactiveidle()` to stabilize the animation state machine, and a retry loop for resource downloads on flaky networks. One costly lesson: leaving the SDK connected while idle continuously burns platform credits, so explicit disconnect logic is mandatory.

Takeaways

— End-to-end latency from user speech to avatar response is roughly 500ms, broken into server-side speech synthesis, parameter delivery, and local WebGL rendering.

— Calling `interactiveidle()` before `speak()` enables interruptible broadcasting; a 150ms delay afterward prevents mouth-shape glitches during rapid re-broadcasts.

— Failing to call `destroy()` on the SDK instance before creating a new one leaves the old WebGL render loop running, producing double-stacked avatars and screen flicker.

— Leaving the SDK connected while idle continuously drains platform credits, so explicit disconnect logic is required.

— Resource download failures on weak networks get no automatic retry from the SDK; a manual retry loop (up to 3 attempts with a 2-second backoff) is necessary for reliable first connections.

— Avatar model and motion data download once at initialization; subsequent interactions transmit only compact action parameters, keeping bandwidth low.

— Multiple LLM backends and avatar presets can be managed through a single config store persisted to localStorage, allowing model and visual theme to switch together.

Conclusions

Parameter-stream digital humans invert the cost structure of video-streamed ones: server-side GPU rendering per user disappears, replaced by a one-time client-side model download and lightweight parameter sync.

The 500ms latency figure is credible for a Wi-Fi + local Chrome setup, but the breakdown reveals speech synthesis as the dominant bottleneck, not rendering or networking.

An `interactiveidle()` call plus a 150ms delay is a heuristic tuned to specific avatar state machines; it signals that the SDK's animation layer is not fully self-consistent across all avatar types and may require per-avatar calibration.

Concepts & terms

Parameter-stream digital human

A 3D avatar driven by transmitting compact action parameters (mouth angles, head rotation, gesture IDs) to a locally rendered model, as opposed to streaming full video frames. This cuts bandwidth and server cost while enabling real-time interruption.

LAM (Large Action Model)

An underlying model that converts text into synchronized speech, facial expression, and body motion parameters, serving as the bridge between language output and embodied 3D behavior.

Embodied interactive intelligence

A system stack combining multimodal perception, agent scheduling, and 3D rendering so an AI can respond with synchronized speech, expression, and gesture in real time, rather than just text.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗