Artificial Intelligence · Frontend · Backend

How to Build a Full-Duplex Voice Agent with Streaming ASR, LLM, and TTS

By 云浪 · Aug 3, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Adding voice to an agent is a systems-integration problem, not a model problem. The difference between a clunky sequential pipeline and a natural conversation is a few hundred lines of buffering, cancellation, and scheduling code that most tutorials skip.

Summary

Voice interaction for AI agents requires three services to run in lockstep: speech recognition, a streaming LLM, and real-time text-to-speech. A full-stack implementation wires a browser's MediaRecorder to Alibaba Cloud's qwen3-asr-flash model, feeds the recognized text into a LangChain-powered SSE chat stream, and then forwards each sentence fragment over a WebSocket to qwen3-tts-flash-realtime for synthesis. The server acts as a transparent relay, buffering text until a punctuation mark or a 15-character threshold triggers a TTS dispatch.

On the frontend, an AudioContext-based player schedules decoded MP3 chunks with precise start times so playback is seamless across segments. The same streamChat function handles both typed and spoken input, and an AbortController cancels in-flight requests when the user sends a new message. The result is a Doubao-style experience where the agent begins reading a reply aloud before the LLM has finished generating it.

The core engineering problem is not any single API call but the buffering and scheduling logic that keeps ASR, LLM, and TTS from drifting apart. A message queue inside the TTS WebSocket handler holds text until the DashScope session is ready, and the client-side player tracks scheduled versus finished segments to know when playback is truly done.

Takeaways

— Alibaba Cloud's qwen3-tts-flash-realtime model accepts streaming text over a WebSocket and returns base64 MP3 chunks as they are synthesized.

— The server_commit mode lets the cloud model decide sentence boundaries automatically; the client just keeps appending text.

— Express's default 100 KB JSON body limit breaks on any audio recording longer than a couple of seconds; bump it to 50 MB for ASR uploads.

— An AudioContext player with source.start(nextStartTime) achieves gapless playback across TTS chunks without gaps or clicks.

— A punctuation-and-length-based dispatch strategy (send on [。！？!?\n] or every 15 characters) lets TTS start reading before the LLM finishes.

— Both voice and text input funnel into the same streamChat function, so behavior is identical regardless of input mode.

— AbortController cancels an in-flight SSE stream when the user sends a new message, preventing stale text from mixing into the current reply.

Conclusions

Most voice-agent demos treat ASR, LLM, and TTS as sequential steps. The real product experience comes from running them as overlapping streams, which turns the server into a timing-sensitive relay rather than a simple proxy.

The 15-character fallback threshold is a pragmatic hack that acknowledges punctuation alone is unreliable for real-time TTS dispatch, especially with LLMs that generate run-on sentences or code blocks.

Using server_commit mode offloads sentence-boundary detection to the cloud model, but it also means the client loses control over when audio is emitted—a trade-off that matters if you need word-level synchronization with animations or transcripts.

Concepts & terms

ASR (Automatic Speech Recognition)

Also called STT (Speech-to-Text). Converts spoken audio into text. Non-streaming ASR waits for a complete utterance; streaming ASR processes audio as it arrives.

TTS (Text-to-Speech)

Converts text into spoken audio. Real-time TTS uses a WebSocket to accept streaming text input and return audio chunks incrementally, rather than waiting for the full text.

server_commit mode

A TTS segmentation strategy where the cloud service automatically detects sentence boundaries from punctuation and semantics. The client only appends text; it never explicitly marks where synthesis should begin.

ITN (Inverse Text Normalization)

A post-processing step in ASR that converts spoken number forms into written numerals—for example, turning 'one hundred twenty-three' into '123'.

SSE (Server-Sent Events)

A unidirectional HTTP-based protocol where the server pushes a stream of text events to the client. Used here to stream LLM tokens as they are generated, with each chunk wrapped in a `data:` line.

AudioContext scheduling

The Web Audio API's AudioContext has an internal clock. Calling source.start(time) with a future timestamp schedules playback precisely, enabling seamless concatenation of multiple audio buffers without gaps.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗