How to Build a Full-Duplex Voice Agent with Streaming ASR, LLM, and TTS

highlight: a11y-dark

We commonly use Agents that have voice functionality, for example when you use Doubao:

Voice input in progress:

You can also switch voices:

This kind of voice input is converted to text, and the large model's response is read aloud via voice, with switchable voices. It's basically an essential feature for common Agents.

To implement this kind of voice interaction, there are two indispensable links: one is speech-to-text (ASR), and the other is text-to-speech (TTS).

ASR (Automatic Speech Recognition), sometimes also called STT (Speech To Text), refers to the same thing, just with different names. TTS (Text To Speech) — text-to-speech.

In this article, we will implement the same voice interaction functionality as Doubao.

We'll use Alibaba Cloud's voice services here; actually, the usage is similar across different providers.

Text-to-Speech (TTS) Implementation

Non-Real-Time Speech Synthesis

Non-real-time speech synthesis doesn't meet the requirements here, because when we use Doubao, we can see that during voice reading, the voice starts reading before the text output is complete; it reads aloud while the text is being output.

Non-real-time speech synthesis, on the other hand, requires waiting for the large model's response to be fully output, i.e., the text is completely output before the voice starts playing. However, non-real-time speech synthesis is a good warm-up project for learning real-time speech synthesis later.

Create a project

mkdir tts-normal
cd tts-normal
npm init -y

We'll use Alibaba Cloud's Qwen series speech synthesis large model qwen3-tts-flash. In fact, the usage of large models from various cloud service providers is similar.

After obtaining the API Key on the Alibaba Cloud Bailian API Key Management page, we can call the Qwen series large models.

Install the required packages:

pnpm install dotenv

Create a .env configuration file:

DASHSCOPE_API_KEY=Your API Key

Because in non-streaming mode, the Qwen series large model response contains a url field pointing to the synthesized audio file. You can use fetch to request this url to get the synthesized audio.

Create tts-normal.mjs

import fs from "fs";
import dotenv from "dotenv";

dotenv.config(); // Load environment variables from .env

const API_KEY = process.env.DASHSCOPE_API_KEY;
const OUTPUT_FILE = "output.mp3"; // Local save path

// Configuration area
const config = {
  model: "qwen3-tts-flash",
  // Voice
  voice: "Serena",
  text: "那我来给大家推荐一款T恤, 这款呢真的是超级好看, 这个颜色呢很显气质, 而且呢也是搭配的绝佳单品, 大家可以闭眼入, 真的是非常好看, 对身材的包容性也很好, 不管啥身材的宝宝呢, 穿上去都是很好看的。推荐宝宝们下单哦。",
};

// Core logic
async function generateTTS() {
  try {
    // 1. Call the multimodal generation API (non-real-time)
    const response = await fetch(
      "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation",
      {
        method: "POST",
        headers: {
          Authorization: `Bearer ${API_KEY}`,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          model: config.model,
          input: {
            text: config.text,
            voice: config.voice,
          },
        }),
      }
    );

    const data = await response.json();

    // 2. Extract the audio file URL (multimodal generation API returns output.audio.url)
    const audioUrl = data?.output?.audio?.url;
    if (!audioUrl) {
      console.error("API Response:", JSON.stringify(data, null, 2));
      throw new Error("API returned no audio.url");
    }

    console.log(`Got audio URL: ${audioUrl}`);

    // 3. Download audio and save as local MP3
    const audioResponse = await fetch(audioUrl);
    const audioBuffer = Buffer.from(await audioResponse.arrayBuffer());
    fs.writeFileSync(OUTPUT_FILE, audioBuffer);

    console.log(`MP3 saved to: ${OUTPUT_FILE}`);
    console.log(`Audio link validity: 24 hours (expired links need to re-call the API)`);
  } catch (error) {
    console.error("Synthesis failed:", error.message);
    throw error;
  }
}

// Execute
generateTTS();

When using fetch to download a file, convert the raw binary data (MP3 audio byte stream) returned by the server into a buffer and write it to a file.

Voice names can be found here: https://help.aliyun.com/zh/model-studio/qwen-tts-voice-list

Execute the node tts-normal.mjs command in the terminal to generate the audio file:

Real-Time Speech Synthesis

The non-real-time speech synthesis method above, which passes the entire text at once to generate speech, doesn't fit our scenario.

For example, Doubao returns responses in a streaming manner, and the voice is also played in a streaming manner.

This requires a real-time speech synthesis interface, which uses WebSocket. Through the WebSocket protocol, text is converted into natural speech in real-time, supporting streaming input and output.

Create a project

mkdir realtime-tts
cd realtime-tts
npm init -y

Install the required packages:

pnpm install dotenv ws

Same as non-real-time speech synthesis, create a .env configuration file:

DASHSCOPE_API_KEY=Your API Key

Load the API Key from environment variables.

Create realtime-tts.mjs

import WebSocket from "ws";
import fs from "fs";
import dotenv from "dotenv";

dotenv.config();

// Configuration area
const config = {
  apiKey: process.env.DASHSCOPE_API_KEY,
  // Real-time speech synthesis model
  model: "qwen3-tts-flash-realtime",
  voice: "Cherry",
  // Text to synthesize (simulating streaming input)
  textChunks: [
    "我不去想是否能够成功",
    "既然选择了远方，便只顾风雨兼程",
    "我不去想能否赢得爱情",
    "既然钟情于玫瑰，就勇敢地吐露真诚",
    "我不去想身后会不会袭来寒风冷雨",
    "既然目标是地平线，留给世界的只能是背影",
    "我不去想未来是平坦还是泥泞",
    "只要热爱生命，一切，都在意料之中",
  ],
  // Singapore region requires replacing WorkspaceId, Beijing region uses wss://dashscope.aliyuncs.com/api-ws/v1/realtime
  wsUrl: "wss://dashscope.aliyuncs.com/api-ws/v1/realtime",
};

// 1. Establish WebSocket connection
const ws = new WebSocket(`${config.wsUrl}?model=${config.model}`, {
  headers: {
    Authorization: `Bearer ${config.apiKey}`,
  },
});

// 2. Handle connection open event
ws.on("open", () => {
  console.log("WebSocket connection established");

  // Send session.update to configure session parameters
  const sessionUpdate = {
    type: "session.update",
    session: {
      mode: "server_commit", // Server intelligently determines segmentation and synthesis timing
      voice: config.voice,
      response_format: "mp3", // Supports mp3, pcm, wav, etc.
    },
  };
  ws.send(JSON.stringify(sessionUpdate));

  // Simulate streaming text sending
  sendTextStream();
});

// 3. Simulate streaming text sending
function sendTextStream() {
  let index = 0;
  const interval = setInterval(() => {
    if (index < config.textChunks.length) {
      const text = config.textChunks[index];
      console.log(`Sending text: ${text}`);

      // Append text to buffer
      ws.send(
        JSON.stringify({
          type: "input_text_buffer.append",
          text: text,
        })
      );
      index++;
    } else {
      clearInterval(interval);
      // Text sending complete, notify server to finish
      ws.send(JSON.stringify({ type: "session.finish" }));
      console.log("Text sending complete, waiting for final audio generation...");
    }
  }, 200); // Send a segment every 200ms, simulating a typewriter effect
}

// 4. Handle server-side return events
ws.on("message", (data) => {
  const response = JSON.parse(data.toString());

  switch (response.type) {
    case "session.created":
      console.log(`Session created, Session ID: ${response.session.id}`);
      break;

    case "response.audio.delta":
      // Receive audio stream data (Base64 encoded)
      const audioBuffer = Buffer.from(response.delta, "base64");
      // In a real project, this could be pushed to a frontend player or written to a file
      fs.appendFileSync("realtime_output.mp3", audioBuffer);
      process.stdout.write("."); // Simple progress indicator
      break;

    case "response.done":
      console.log("\nSingle audio response complete");
      break;

    case "session.finished":
      console.log("Real-time speech synthesis fully complete!");
      ws.close();
      break;

    default:
      // Handle other events or errors
      if (response.type?.includes("error")) {
        console.error("Server error:", response);
      }
      break;
  }
});

// 5. Error and close handling
ws.on("error", (err) => console.error("WebSocket error:", err.message));
ws.on("close", (code, reason) =>
  console.log(`Connection closed (Code: ${code}, Reason: ${reason || "Normal"})`)
);

Alibaba Cloud's Qwen real-time speech synthesis API offers two interaction modes (sentence segmentation strategies): server_commit mode and commit mode.

In server_commit mode, the server intelligently handles text segmentation and synthesis timing. The client only needs to continuously append text, without worrying about segmentation and submission.

In commit mode, the client actively submits the text buffer to trigger synthesis.

Here we use server_commit mode. Because in the code above, the text is sent in chunks in a streaming manner—8 sentences sent every 200ms—the client doesn't care about and shouldn't judge where the semantic boundaries of each sentence are. server_commit lets the cloud model automatically determine "this sentence is finished, synthesis can begin" based on punctuation and semantics. You just keep feeding it text—simpler and more natural.

The client (Node.js code here) communicates with the Alibaba Cloud real-time speech synthesis WebSocket server through events.

Events are divided into client events and server events.

Client events are initiated by the client, sending data to the server.

Server events are initiated by the server, sending data to the client.

The client events used here are session.update and input_text_buffer.append.

The session.update event is used to tell the server, before speech synthesis begins, how you want the synthesis to be done—what voice, what format, what segmentation strategy. If not sent, the system will use default configurations.

ws.send(
  JSON.stringify({
    type: "session.update", // Event type: update session configuration
    session: {
      mode: "server_commit", // Segmentation strategy: let the server automatically judge
      voice: "Cherry", // Voice
      response_format: "mp3", // Output format
    },
  })
);

The input_text_buffer.append event is used to append text to be synthesized to the text buffer. In server_commit mode, the text is appended to the server's text buffer; in commit mode, the text is appended to the client's text buffer.

The server events used here are session.created, response.audio.delta, response.done, and session.finished.

The session.created event is the first event sent by the server after the client connects. This event returns the server's default configuration information for this connection.

The response.audio.delta event is sent to the client when the model incrementally generates new audio data. Audio data can be obtained in this event.

The response.done event is sent by the server to the client when a single response generation is complete.

The session.finished event is sent by the server to the client when all response generation is complete. It signifies the end of the entire session, and the connection can be closed.

Final running effect:

You can see that the audio is not generated completely at once, but segment by segment. This is the effect of real-time speech synthesis.

Speech-to-Text (ASR) Implementation

Next, let's try speech recognition ASR (Automatic Speech Recognition), also called STT (Speech To Text) in some places.

This doesn't need to be streaming. When we use Doubao, the text is usually converted after we finish speaking a sentence.

Create a project

mkdir asr-offline
cd asr-offline
npm init -y

Install the required packages:

pnpm install dotenv

Same as the speech synthesis above, create a .env file and fill in the large model's API key:

DASHSCOPE_API_KEY=Your API Key

Create asr-offline.mjs

import fs from "fs";
import path from "path";
import dotenv from "dotenv";

dotenv.config();

const API_KEY = process.env.DASHSCOPE_API_KEY;
const AUDIO_FILE = path.resolve("./realtime_output.mp3");

async function transcribeLocalAudio() {
  // Check if local file exists
  if (!fs.existsSync(AUDIO_FILE)) {
    console.error(`Local audio file not found: ${AUDIO_FILE}`);
    return;
  }

  try {
    console.log("Recognizing local audio file...");

    // 1. Read audio file and convert to base64
    const audioBuffer = fs.readFileSync(AUDIO_FILE);
    const audioBase64 = audioBuffer.toString("base64");
    // Construct data URL (MP3 format)
    const audioDataUrl = `data:audio/mp3;base64,${audioBase64}`;

    // 2. Call multimodal generation API, passing audio data URL in messages format
    const response = await fetch(
      "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation",
      {
        method: "POST",
        headers: {
          Authorization: `Bearer ${API_KEY}`,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          model: "qwen3-asr-flash",
          input: {
            messages: [
              {
                role: "user",
                content: [
                  {
                    audio: audioDataUrl,
                  },
                ],
              },
            ],
          },
          parameters: {
            enable_itn: true, // Enable ITN, converting Chinese numerals (like "一百") to Arabic numerals (100)
          },
        }),
      }
    );

    if (!response.ok) {
      console.error(`HTTP Error: ${response.status} ${response.statusText}`);
      const errorBody = await response.text();
      console.error("Response content:", errorBody);
      return;
    }

    const data = await response.json();

    // 3. Extract recognition result
    // Response structure: output.choices[0].message.content[0].text
    const text = data?.output?.choices?.[0]?.message?.content?.[0]?.text;

    if (text) {
      console.log("\n========== Recognition Result ==========");
      console.log(text);
      console.log("======================================\n");
    } else {
      console.error(
        "No recognition result obtained, full response:",
        JSON.stringify(data, null, 2)
      );
    }
  } catch (error) {
    console.error("Speech recognition failed:", error.message);
  }
}

transcribeLocalAudio();

We use the audio file realtime_output.mp3 generated in the previous real-time speech synthesis section as the audio to be recognized by this script:

You can see the audio is correctly recognized.

Integrated Implementation

Next, we start implementing the same voice interaction as Doubao. The specific interaction flow is:

Use the browser's native MediaRecorder API to record audio
Call the speech-to-text interface to convert the recorded audio to text
Pass the generated text to the large model's response interface
Pass the text generated by the large model's response interface to the text-to-speech interface for voice playback

Server-Side Implementation

The server mainly implements three interfaces:

SSE streaming chat interface
Speech-to-text interface (ASR)
Streaming speech synthesis (TTS) WebSocket interface

SSE Streaming Chat Interface Implementation

app.post("/api/chat", handleChatStream); // Support POST requests
app.get("/api/chat", handleChatStream); // Support GET requests

Because the browser's native EventSource only supports GET requests, GET requests are additionally supported here. However, due to the limitations of GET requests:

Parameters can only be placed in the URL
URL length is limited
Parameters are exposed in the URL
Not suitable for sending complex JSON

Therefore, in actual AI applications, we generally use the POST request chat interface, using fetch + ReadableStream to get SSE streaming data, rather than using EventSource.

Specific implementation of the handleChatStream streaming chat interface:

import { ChatOpenAI } from "@langchain/openai";

async function handleChatStream(req, res) {
  // Step 1: Get user message
  const message =
    req.query.message || // GET: /api/chat?message=Hello
    (req.body && req.body.message) || // POST: from body
    "Hello"; // Default fallback value

  // Step 2: Set SSE response headers
  res.setHeader("Content-Type", "text/event-stream"); // Tell browser this is an SSE stream
  res.setHeader("Cache-Control", "no-cache"); // Disable caching
  res.setHeader("Connection", "keep-alive"); // Keep connection alive
  res.setHeader("X-Accel-Buffering", "no"); // Disable nginx buffering
  res.flushHeaders(); // Send response headers immediately, don't wait for body

  // Step 3: Create large model instance
  const model = new ChatOpenAI({
    model: process.env.MODEL_NAME || "qwen-plus",
    apiKey: process.env.DASHSCOPE_API_KEY,
    configuration: {
      baseURL: process.env.OPENAI_BASE_URL,
    },
    streaming: true, // Key: Enable streaming mode
    temperature: 0.7, // Control creativity (0=precise, 1=creative)
  });

  // Step 4: Stream generate and push chunk by chunk
  try {
    const stream = await model.stream(message);

    for await (const chunk of stream) {
      // chunk could be one character, a few characters, or a word
      const content = chunk.content;
      if (content) {
        // Format as SSE: data: {"content":"You"}\n\n
        const sseData = `data: ${JSON.stringify({ content })}\n\n`;
        // Use Buffer.from to ensure UTF-8 encoding, avoiding Chinese garbled characters
        res.write(Buffer.from(sseData, "utf-8"));
      }
    }

    // Step 5: Send end marker
    res.write(Buffer.from("data: [DONE]\n\n", "utf-8"));
    res.end(); // Close connection
  } catch (error) {
    // Error handling
    // Case 1: Response headers not yet sent → return normal JSON error
    if (!res.headersSent) {
      return res.status(500).json({ error: error.message });
    }
    // Case 2: Already started streaming output → send error in SSE format
    res.write(
      Buffer.from(
        `data: ${JSON.stringify({ error: error.message })}\n\n`,
        "utf-8"
      )
    );
    res.write(Buffer.from("data: [DONE]\n\n", "utf-8"));
    res.end();
  }
}

Speech-to-Text Interface (ASR) Implementation

The implementation of the speech-to-text interface is relatively simple. The specific process is: receive the base64 audio sent by the frontend, forward it to the Alibaba Cloud speech recognition service, recognize the text, and finally return the recognized text to the frontend.

Express's default JSON body size limit is only 100KB, while the base64 encoding of a 5-second recording can reach 200KB+. Therefore, Express's default JSON body size limit needs to be adjusted to 50MB:

app.use(express.json({ limit: "50mb" }));

50MB is enough to record about 2 hours of audio, which is more than enough for conversation scenarios.

Specific implementation of the speech-to-text interface:

app.post("/api/asr", async (req, res) => {
  try {
    // 1. Parameter validation
    const { audio } = req.body;
    if (!audio) {
      return res.status(400).json({ error: "Missing audio field" });
    }

    console.log("Calling speech recognition...");

    // 2. Call Alibaba Cloud ASR API
    const response = await fetch(
      "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation",
      {
        method: "POST",
        headers: {
          Authorization: `Bearer ${process.env.DASHSCOPE_API_KEY}`,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          model: "qwen3-asr-flash",
          input: {
            messages: [
              {
                role: "user",
                content: [{ audio }],
              },
            ],
          },
          parameters: {
            enable_itn: true,
          },
        }),
      }
    );

    const data = await response.json();

    // 3. Error handling
    if (!response.ok) {
      console.error("ASR API Error:", data);
      return res
        .status(response.status)
        .json({ error: data.message || "Speech recognition failed" });
    }

    // 4. Extract result
    const text = data?.output?.choices?.[0]?.message?.content?.[0]?.text;

    if (text) {
      console.log("Recognition result:", text);
      res.json({ text });
    } else {
      console.error("No recognition result obtained:", JSON.stringify(data));
      res.status(500).json({ error: "No recognition result obtained" });
    }
  } catch (error) {
    // 5. Fallback exception
    console.error("ASR Interface Error:", error.message);
    res.status(500).json({ error: error.message });
  }
});

enable_itn parameter:

enable_itn: true

ITN = Inverse Text Normalization, converts Chinese numerals in recognition results to Arabic numerals:

Original: "价格是一百二十三块五毛"
ITN:  "价格是123.5元"

This is a parameter specific to Alibaba Cloud ASR, not something all standard speech recognition services have.

Streaming Speech Synthesis (TTS) WebSocket Interface Implementation

This is the most technically challenging interface on the server side.

The Node.js server acts as a relay, forwarding the text sent by the browser to the Alibaba Cloud real-time speech synthesis service, and then forwarding the synthesized audio back to the browser for playback. The entire process uses WebSocket, with data being transmitted as it goes, no waiting required.

The ws npm package is used here. It is the most popular WebSocket implementation in the Node.js community, providing both server and client APIs.

import { WebSocketServer, WebSocket } from "ws";

WebSocketServer acts as a server to accept browser connections
WebSocket acts as a client to connect to DashScope (Alibaba Cloud speech synthesis service)

// Create HTTP server (shared by Express + WebSocket)
const server = http.createServer(app);
const wss = new WebSocketServer({ server, path: "/ws/tts" });

new WebSocketServer creates a WebSocket server instance. Its core function is to enable the Node.js server to accept and handle browser WebSocket connections. It upgrades an ordinary HTTP server to an HTTP + WebSocket dual-protocol server.

The server option is used to mount the WebSocketServer onto an existing HTTP server, allowing it to share the same port with the HTTP service, without needing to open a separate port.

path specifies the websocket interface path as /ws/tts.

When a browser initiates a WebSocket connection, the connection event is triggered. In the connection event, we can get the browser's WebSocket connection instance and start processing data.

On one hand, it receives messages from the browser; on the other hand, it creates a WebSocket client to connect to the Alibaba Cloud real-time speech synthesis service, forwards the text messages received from the browser to the Alibaba Cloud real-time speech synthesis server, and after getting the synthesized audio data, uses the browser's WebSocket connection instance to send the audio data to the browser.

Specific implementation of the streaming speech synthesis (TTS) WebSocket interface:

/**
 * Streaming Speech Synthesis (TTS) WebSocket Interface
 * Path: ws://localhost:3000/ws/tts
 *
 * Browser → Server:
 *   { type: "text", data: "Content to synthesize" }  Append text
 *   { type: "finish" }                              Text ended, wait for final audio
 *
 * Server → Browser:
 *   { type: "audio", data: "base64 mp3..." }  Audio chunk
 *   { type: "done" }                          Synthesis complete
 *   { type: "error", message: "..." }         Error
 */
wss.on("connection", (browserWs) => {
  console.log("TTS WebSocket client connected");

  // Reference to the Alibaba Cloud WebSocket connection
  let dashScopeWs = null;
  // Flag indicating whether the browser has sent a finish
  let isFinished = false;
  // Flag indicating whether the DashScope session is created
  let sessionReady = false;
  // Cache messages before the session is ready
  let pendingMessages = [];

  const ttsUrl = `wss://dashscope.aliyuncs.com/api-ws/v1/realtime?model=qwen3-tts-flash-realtime`;

  // Real-time speech synthesis reference: https://help.aliyun.com/zh/model-studio/interactive-process-of-qwen-tts-realtime-synthesis?spm=a2c4g.11186623.help-menu-2400256.d_2_5_1_1_0.158c29ee8q62gF&scm=20140722.H_2963385._.OR_help-T_cn~zh-V_1
  // Connect to Alibaba Cloud Bailian real-time TTS
  dashScopeWs = new WebSocket(ttsUrl, {
    headers: {
      Authorization: `Bearer ${process.env.DASHSCOPE_API_KEY}`,
    },
  });

  dashScopeWs.on("open", () => {
    console.log("Connected to DashScope TTS");

    // Configure session: server_commit mode lets the server automatically determine segmentation and synthesis timing
    // Set voice, format, mode, etc. by sending the session.update event.
    dashScopeWs.send(
      JSON.stringify({
        type: "session.update",
        session: {
          mode: "server_commit",
          voice: "Cherry",
          response_format: "mp3",
        },
      })
    );
  });

  // Receive messages from Alibaba Cloud real-time speech synthesis service
  dashScopeWs.on("message", (data) => {
    const msg = JSON.parse(data.toString());
    console.log(`DashScope: ${msg.type}`);

    switch (msg.type) {
      case "session.created":
        console.log(`TTS Session created: ${msg.session?.id}`);
        // Session ready, send cached messages
        sessionReady = true;
        if (pendingMessages.length > 0) {
          console.log(`Sending ${pendingMessages.length} cached messages`);
          for (const pending of pendingMessages) {
            dashScopeWs.send(JSON.stringify(pending));
          }
          pendingMessages = [];
        }
        break;

      case "response.audio.delta":
        // Forward audio chunk to browser
        if (browserWs.readyState === WebSocket.OPEN) {
          browserWs.send(JSON.stringify({ type: "audio", data: msg.delta }));
        }
        break;

      case "response.done":
        // Single audio segment complete
        break;

      case "session.finished":
        console.log("TTS synthesis fully complete");
        if (browserWs.readyState === WebSocket.OPEN) {
          browserWs.send(JSON.stringify({ type: "done" }));
          browserWs.close();
        }
        break;

      default:
        if (msg.type?.includes("error")) {
          console.error("TTS Error:", msg);
          if (browserWs.readyState === WebSocket.OPEN) {
            browserWs.send(
              JSON.stringify({
                type: "error",
                message: msg.message || "TTS synthesis error",
              })
            );
          }
        }
        break;
    }
  });

  dashScopeWs.on("error", (err) => {
    console.error("DashScope WebSocket Error:", err.message);
    if (browserWs.readyState === WebSocket.OPEN) {
      browserWs.send(JSON.stringify({ type: "error", message: err.message }));
    }
  });

  dashScopeWs.on("close", () => {
    console.log("DashScope TTS connection closed");
  });

  // Receive messages from the browser
  browserWs.on("message", (data) => {
    try {
      const msg = JSON.parse(data.toString());
      console.log(`Received browser message: type=${msg.type}`);

      if (msg.type === "text" && msg.data) {
        // Cache message, send after DashScope session is ready
        const ttsMsg = {
          type: "input_text_buffer.append",
          text: msg.data,
        };
        if (sessionReady && dashScopeWs?.readyState === WebSocket.OPEN) {
          console.log(
            `DashScope: input_text_buffer.append (${msg.data.length} chars)`
          );
          // Forward the text message from the browser to the Alibaba Cloud real-time speech synthesis service to convert text to speech
          dashScopeWs.send(JSON.stringify(ttsMsg));
        } else {
          // Cache message, send after Alibaba Cloud real-time speech synthesis service session is ready
          pendingMessages.push(ttsMsg);
          console.log(
            `Caching message (sessionReady=${sessionReady}, wsState=${dashScopeWs?.readyState})`
          );
        }
      } else if (msg.type === "finish") {
        isFinished = true;
        const finishMsg = { type: "session.finish" };
        if (sessionReady && dashScopeWs?.readyState === WebSocket.OPEN) {
          console.log(`DashScope: session.finish`);
          // Send session.finish event to notify Alibaba Cloud real-time speech synthesis service that no more text input is coming,
          // Alibaba Cloud real-time speech synthesis service will return the remaining audio and then close the connection.
          dashScopeWs.send(JSON.stringify(finishMsg));
        } else {
          pendingMessages.push(finishMsg);
          console.log(
            `Caching finish (sessionReady=${sessionReady}, wsState=${dashScopeWs?.readyState})`
          );
        }
      }
    } catch (err) {
      console.error("Failed to parse browser message:", err.message);
    }
  });

  browserWs.on("close", () => {
    console.log("TTS client disconnected");
    // Clean up DashScope connection
    if (dashScopeWs?.readyState === WebSocket.OPEN) {
      if (!isFinished) {
        dashScopeWs.send(JSON.stringify({ type: "session.finish" }));
      }
      dashScopeWs.close();
    }
  });

  browserWs.on("error", (err) => {
    console.error("Browser WebSocket Error:", err.message);
  });
});

Frontend Implementation

Use MediaRecorder to record audio.

Call the speech-to-text interface to convert the recording to text.

Then pass the text to the large model streaming response interface.

Pass the large model's response to the text-to-speech interface for voice playback.

Microphone Recording Module

This is the most user-interaction-intensive part of the entire application. It involves three events: press to start, release to stop, mouse leave to cancel.

Event Binding

micBtn.addEventListener("mousedown", startRecording);
micBtn.addEventListener("touchstart", (e) => {
  e.preventDefault();
  startRecording();
});
micBtn.addEventListener("mouseup", stopRecording);
micBtn.addEventListener("touchend", (e) => {
  e.preventDefault();
  stopRecording();
});
micBtn.addEventListener("mouseleave", () => {
  if (isRecording) stopRecording();
});

Supports both PC mouse and mobile touch screen. mouseleave handles the case where the user presses and drags out of the button—dragging out also stops recording, preventing a "deadlock".

e.preventDefault() in touchstart prevents the mobile browser's default behavior (like page zoom, long-press menu).

Start Recording

async function startRecording() {
  if (isProcessing || isRecording) return; // Prevent re-entry

  // Pop up browser permission dialog, request microphone access
  stream = await navigator.mediaDevices.getUserMedia({ audio: true });

  const mimeType = MediaRecorder.isTypeSupported("audio/webm;codecs=opus")
    ? "audio/webm;codecs=opus" // Best for Chrome/Firefox
    : MediaRecorder.isTypeSupported("audio/webm")
    ? "audio/webm" // Fallback
    : "audio/mp4"; // Safari fallback

  mediaRecorder = new MediaRecorder(stream, { mimeType });
  audioChunks = [];
  // Automatically triggered when MediaRecorder produces a usable chunk of media data during recording
  mediaRecorder.ondataavailable = (e) => {
    if (e.data.size > 0) audioChunks.push(e.data);
  };
  // Start recording
  mediaRecorder.start();
  isRecording = true;
  micBtn.classList.add("recording");
}

Stop Recording

async function stopRecording() {
  if (!mediaRecorder || mediaRecorder.state !== "recording") return;
  isRecording = false;
  mediaRecorder.stop(); // Triggers ondataavailable and onstop
  if (stream) {
    stream.getTracks().forEach((t) => t.stop()); // Release microphone
    stream = null;
  }
  micBtn.classList.remove("recording");

  mediaRecorder.onstop = async () => {
    if (audioChunks.length === 0) {
      setStatus("No audio recorded", "");
      return;
    }
    await transcribeAndChat(); // Enter recognition and chat flow
  };
}

stream.getTracks().forEach(t => t.stop()) releases the microphone hardware resource, and the red recording indicator on the browser tab will disappear.

Speech Recognition → Auto Chat Flow

async function transcribeAndChat() {
  isProcessing = true;
  micBtn.disabled = true;
  sendBtn.disabled = true;
  setStatus("🔉 Recognizing speech...", "transcribing");

  // 1. Merge audio chunks into a Blob
  const blob = new Blob(audioChunks, { type: mediaRecorder.mimeType });

  // 2. Blob → base64 Data URL
  const base64 = await blobToBase64(blob);

  // 3. Send to backend ASR interface
  const asrRes = await fetch("/api/asr", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ audio: base64 }),
  });
  const asrData = await asrRes.json();

  // 4. Display recognition result + auto initiate chat
  const recognizedText = asrData.text;
  addMessage("user", recognizedText);
  await streamChat(recognizedText); // Enter streaming chat
}

The entire flow is a linear asynchronous chain:

Recording stopped → Blob → base64 → POST /api/asr → Get text
                                               ↓
                                    Display user bubble + streamChat()

Text Sending

async function sendTextMessage() {
  const text = textInput.value.trim();
  if (!text || isProcessing) return;

  textInput.value = ""; // Clear input box
  textInput.style.height = "auto"; // Reset height
  isProcessing = true;
  addMessage("user", text);
  await streamChat(text); // ← Same chat entry point
  isProcessing = false;
  textInput.focus();
}

Both speech recognition and text sending ultimately go to the same streamChat() function. This ensures the behavior of the two input methods is completely consistent.

Auto-expanding input box:

textInput.addEventListener("input", () => {
  textInput.style.height = "auto";
  textInput.style.height = Math.min(textInput.scrollHeight, 100) + "px";
});

First reset to auto to get the real content height, then set it but not exceeding 100px, achieving a textarea with adaptive height.

Streaming Chat Core: `streamChat()`

This is the most core, largest code volume function in the entire file. It does three things simultaneously: streaming receive AI reply + render to bubble + forward text to TTS.

Cancel Old Request

if (activeAbortController) {
  activeAbortController.abort(); // Cancel previous fetch
  activeAbortController = null;
}
const abortController = new AbortController();
activeAbortController = abortController;

When the user quickly sends multiple messages in succession, the previous one is canceled by the new request before it finishes returning. If not canceled, data from the old request will continue to be written into the bubble, mixing with the new message.

Initiate Chat Request

const response = await fetch("/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ message }),
  signal: abortController.signal, // Bind cancel signal
});

Read SSE Stream Chunk by Chunk

const reader = response.body.getReader();
const decoder = new TextDecoder("utf-8");
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop() || ""; // Keep the incomplete last line

  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const data = line.slice(6); // Remove "data: " prefix
    // ...
  }
}

Every step of this loop has its purpose:

Step	Code	Purpose
Read	`await reader.read()`	Wait for the server to push the next chunk of data, async suspension
Decode	`decoder.decode(value, { stream: true })`	`stream: true` tells the decoder this is streaming data, preventing multi-byte UTF-8 characters from being cut off in the middle
Split lines	`buffer.split("\n")`	Split by single newline, ensuring each complete line can be processed immediately
Cache	`buffer = lines.pop()`	The last line might be incomplete, keep it for the next concatenation

Handling Three Data Types

// 1. End marker
if (data === "[DONE]") {
  cleanup();
  // Send TTS tail buffer + end signal
  return;
}

// 2. Error message
const parsed = JSON.parse(data);
if (parsed.error) {
  currentAssistantMsg.textContent = `${parsed.error}`;
  cleanup();
  return;
}

// 3. Normal content
if (parsed.content) {
  fullContent += parsed.content;
  currentAssistantMsg.textContent = fullContent; // Full text replacement
  messagesEl.scrollTop = messagesEl.scrollHeight; // Auto scroll to bottom

  // TTS related...
}

Why use textContent instead of innerHTML? To prevent XSS attacks—if the AI reply contains HTML tags, textContent will display them as-is without executing. It also preserves white-space: pre-wrap to display line breaks normally.

Why replace with the full text fullContent each time? Instead of += appending? Because AI replies might contain Markdown or special formatting, accumulating to a complete string before unified rendering is safer. The current implementation is plain text, so fullContent += then full assignment has the same effect as direct +=, but the former leaves room for future expansion.

Generate While Reading Aloud—Real-time Linkage of Streaming Chat and TTS

if (ttsWs && ttsWs.readyState === WebSocket.OPEN) {
  ttsBuffer += parsed.content;
  // Send when encountering period, question mark, exclamation mark, newline, or when buffer is long enough
  if (
    /[。！？!?\n]/.test(parsed.content) ||
    ttsBuffer.length >= TTS_CHUNK_SIZE
  ) {
    ttsWs.send(JSON.stringify({ type: "text", data: ttsBuffer }));
    ttsBuffer = "";
  }
}

Segmentation strategy: Don't wait until the entire paragraph is finished to send to TTS. Instead, send when encountering punctuation or accumulating 15 characters. This way, TTS can start reading the previous sentences while the AI is still generating subsequent text, achieving a "speaking while thinking" effect.

Error Handling

} catch (err) {
  if (err.name === "AbortError") {
    console.log("Request canceled");  // Silent handling
    return;
  }
  // Real network error
  cleanup();
  if (!fullContent) currentAssistantMsg.textContent = "Connection failed";
}

Distinguishing AbortError from real network errors is necessary. AbortError is triggered by our active abort() call, not indicating a problem—a user sending a new message to cancel an old request is normal behavior and should not display "Connection failed".

TTS WebSocket Client

Instantiate a WebSocket client to receive audio data.

function connectTTS(audioPlayer, assistantMsg) {
  const wsUrl = `${location.protocol === "https:" ? "wss:" : "ws:"}//${location.host}/ws/tts`;
  const ws = new WebSocket(wsUrl);

Protocol adaptation: Automatically choose ws:// or wss:// based on whether the current page is HTTP or HTTPS.

Three Types of Received Messages

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);

  if (msg.type === "audio" && msg.data) {
    audioPlayer.enqueue(msg.data); // Audio chunk enqueued for playback
  } else if (msg.type === "done") {
    audioPlayer.onAllDone = () => {
      setStatus("", "");
      assistantMsg.classList.remove("speaking");
    };
    audioPlayer.waitForFinish(); // Wait for queue to finish playing
    ws.close();
  } else if (msg.type === "error") {
    setStatus("", "");
    ws.close();
  }
};

msg.data is base64 encoded MP3 data. It is passed directly to AudioPlayer, which handles decoding and playback.

AudioPlayer: Streaming Audio Player Based on AudioContext

class AudioPlayer {
  constructor() {
    this.ctx = null;           // AudioContext instance
    this.queue = [];           // Queue of audio segments to play
    this.nextStartTime = 0;    // Start time for the next segment
    this.scheduledCount = 0;   // Number of segments scheduled
    this.finishedCount = 0;    // Number of segments finished playing
  }

Why use AudioContext instead of the `<audio>` tag?

	`<audio>` tag	AudioContext
Multi-segment continuous playback	Need to listen to `ended` event and manually switch, causing gaps	`source.start(time)` precise scheduling, seamless connection
Decoding	Browser automatic	Manual `decodeAudioData`
Flexibility	Low	High (can add sound effects, adjust volume, etc.)

Core Methods: `enqueue` + `_playNext`

enqueue(base64Data) {
  this._ensureContext();           // Create AudioContext on first use
  this.queue.push(base64Data);
  if (!this.isPlaying_) {
    this.isPlaying_ = true;
    this._playNext();             // If not playing, start immediately
  }
}

async _playNext() {
  const base64Data = this.queue.shift();
  // base64 → Uint8Array → AudioBuffer
  const binaryStr = atob(base64Data);
  const bytes = new Uint8Array(binaryStr.length);
  for (let i = 0; i < binaryStr.length; i++) {
    bytes[i] = binaryStr.charCodeAt(i);
  }
  const audioBuffer = await this.ctx.decodeAudioData(bytes.buffer);

  // Create audio source and schedule playback
  const source = this.ctx.createBufferSource();
  source.buffer = audioBuffer;
  source.connect(this.ctx.destination);
  source.start(this.nextStartTime);  // Key: specify start time
  this.nextStartTime += audioBuffer.duration;
}

source.start(this.nextStartTime) is the key to seamless connection. AudioContext has its own internal clock, and start(time) tells it to start playing at a precise future moment. Even if the next audio segment hasn't finished decoding before the previous one ends, as long as nextStartTime is calculated correctly, there will be no gap.

Completion Detection

source.onended = () => {
  this.finishedCount++;
  this._checkAllDone();  // All segments finished playing?
};

_checkAllDone() {
  if (this.queue.length === 0
      && this.finishedCount >= this.scheduledCount
      && this.onAllDone) {
    this.onAllDone();  // Callback: reset status bar, remove speaking style
  }
}

Because audio is decoded and played asynchronously, determining "all played" requires simultaneously meeting three conditions: queue is empty, scheduled count equals finished count, and callback is registered.

Important Helper Functions

blobToBase64 function:

function blobToBase64(blob) {
  return new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.onloadend = () => resolve(reader.result);
    reader.onerror = reject;
    reader.readAsDataURL(blob);
  });
}

readAsDataURL completes binary reading and base64 encoding in one step, with the result format being "data:audio/webm;base64,AAAA...". This is exactly the format required for the audio field in the backend /api/asr interface.

Effect Demonstration

Code uploaded to github: https://github.com/Panda-plus580/demo

Readers can also try to implement it themselves.

Summary

This article implemented the same voice interaction as Doubao.

Technically, using a Node.js server as a transparent proxy, it connects the complete data chain of frontend MediaRecorder recording → Alibaba Cloud ASR recognition → LangChain SSE streaming dialogue → WebSocket real-time TTS synthesis → AudioContext seamless playback, and achieves a parallel experience of "AI is still thinking, voice has already started reading" through a punctuation segmentation + 15-character fallback strategy.

The core challenge of the entire voice interaction implementation lies in how to design the forwarding, buffering, and scheduling mechanisms for streaming data, so that the three asynchronous links of ASR, LLM, and TTS mesh together like precision gears.

References

Non-Real-Time Speech Synthesis

Real-Time Speech Synthesis

Client Events

Server Events

Non-Real-Time Speech Recognition

How to Build a Full-Duplex Voice Agent with Streaming ASR, LLM, and TTS

highlight: a11y-dark

Text-to-Speech (TTS) Implementation

Non-Real-Time Speech Synthesis

Real-Time Speech Synthesis

Speech-to-Text (ASR) Implementation

Integrated Implementation

Server-Side Implementation

SSE Streaming Chat Interface Implementation

Speech-to-Text Interface (ASR) Implementation

Streaming Speech Synthesis (TTS) WebSocket Interface Implementation

Frontend Implementation

Microphone Recording Module

Event Binding

Start Recording

Stop Recording

Speech Recognition → Auto Chat Flow

Text Sending

Streaming Chat Core: streamChat()

Cancel Old Request

Initiate Chat Request

Read SSE Stream Chunk by Chunk

Handling Three Data Types

Generate While Reading Aloud—Real-time Linkage of Streaming Chat and TTS

Error Handling

TTS WebSocket Client

Three Types of Received Messages

AudioPlayer: Streaming Audio Player Based on AudioContext

Why use AudioContext instead of the <audio> tag?

Core Methods: enqueue + _playNext

Completion Detection

Important Helper Functions

Effect Demonstration

Summary

References

Streaming Chat Core: `streamChat()`

Why use AudioContext instead of the `<audio>` tag?

Core Methods: `enqueue` + `_playNext`