跪拜 Guibai
← Back to the summary

The Byte-by-Byte Pipeline That Powers Every AI Typing Effect

Ordinary HTTP interfaces usually wait until all content is generated before returning the result in one go.

If a large model generates a long article, the user may face a blank page for several seconds or even longer. Even if the model is working normally, the product looks like it's "stuck."

Streaming output changes the response method: every time the model generates a small segment of content, the server immediately sends it to the client, and the page continuously appends these incremental contents to the existing text.

User submits a question
→ Large model generates some tokens
→ Server sends a data event
→ Browser reads and parses
→ Vue updates the page
→ Continue waiting for the next segment of content

This article uses Vue 3 and the Fetch API to break down how the browser reads and parses the data stream returned by a large model.

What does stream: true change?

A non-streaming request returns a complete JSON:

{
  "choices": [
    {
      "message": {
        "content": "Complete answer"
      }
    }
  ]
}

The frontend only needs to wait and parse:

const data = await response.json();
content.value = data.choices[0].message.content;

After enabling streaming output, the response body is no longer a complete object that can be directly response.json(), but a continuously arriving data stream:

data: {"choices":[{"delta":{"content":"你"}}]}

data: {"choices":[{"delta":{"content":"好"}}]}

data: [DONE]

Each delta.content is the newly added content this time, and the frontend needs to read and concatenate it segment by segment.

ReadableStream is a data pipeline in the browser

After Fetch receives a response, response.body is a ReadableStream:

const response = await fetch('/api/stream', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    prompt: question.value,
  }),
});

if (!response.ok) {
  throw new Error(`Request failed: ${response.status}`);
}

if (!response.body) {
  throw new Error('Current browser does not support streaming response');
}

const reader = response.body.getReader();

The request here is to its own /api/stream interface. The browser only submits the question and does not save the API Key of the large model service.

getReader() is like installing a reader on the data pipeline. Each call:

const { value, done } = await reader.read();

will get two values:

value: The binary data read this time
done: Whether the data stream has ended

If there is no new data temporarily, reader.read() will wait; the Promise will only resolve after the server continues to send.

Why can't Uint8Array be displayed directly?

The value returned by reader.read() is usually a Uint8Array.

It stores a set of unsigned integers from 0 to 255, essentially the raw bytes transmitted over the network:

Uint8Array(6) [228, 189, 160, 229, 165, 189]

The browser cannot directly display these numbers as "你好", so TextDecoder is needed for decoding:

const decoder = new TextDecoder('utf-8');

Use streaming decoding when reading data:

const text = decoder.decode(value, {
  stream: true,
});

UTF-8 Chinese characters are usually composed of multiple bytes, and network data can be split at any byte position.

stream: true makes the decoder retain incomplete bytes and continue decoding after the next batch of data arrives, avoiding a Chinese character being split exactly between two read() calls.

After all data is read, call once more:

buffer += decoder.decode();

This flushes out the remaining bytes inside the decoder.

ReadableStream and SSE are not the same thing

These two concepts often appear together, but their responsibilities are different.

ReadableStream: The way the browser reads the response body
SSE: The format for the server to organize text events

SSE is the abbreviation for Server-Sent Events, and the common format is as follows:

data: First piece of data

data: Second piece of data

Each event can contain fields such as event, id, retry, and data, and events are separated by blank lines.

Large model compatible interfaces usually put JSON into the data: field and use [DONE] to indicate the end.

One read() does not equal one SSE event

The network is only responsible for transmitting bytes and does not understand the business boundaries of JSON and SSE.

One reader.read() may get:

Half an SSE event
One complete event
Multiple complete events
The second half of the previous event and the first half of the next event

Therefore, you cannot assume that each value can be directly JSON.parse().

The correct approach is to prepare a buffer:

Append new text to buffer
→ Split complete events by SSE blank lines
→ Keep the last incomplete segment
→ Wait for the next batch of data to continue concatenation

Encapsulate an SSE event parsing function

First, process an already complete SSE event:

function parseSSEEvent(eventText, onDelta) {
  const payload = eventText
    .split(/\r?\n/)
    .filter(line => line.startsWith('data:'))
    .map(line => line.slice(5).trimStart())
    .join('\n');

  if (!payload) return false;

  if (payload === '[DONE]') {
    return true;
  }

  const data = JSON.parse(payload);
  const delta =
    data.choices?.[0]?.delta?.content;

  if (delta) {
    onDelta(delta);
  }

  return false;
}

This function accomplishes three things:

Returning true indicates that the server has sent the end marker.

Complete reading and parsing of the response stream

Next, connect ReadableStream, TextDecoder, and SSE event boundaries:

async function readSSEStream(response, onDelta) {
  if (!response.body) {
    throw new Error('Response body is not a readable stream');
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder('utf-8');
  let buffer = '';
  let finished = false;

  while (!finished) {
    const { value, done } = await reader.read();

    if (done) {
      buffer += decoder.decode();
      break;
    }

    buffer += decoder.decode(value, {
      stream: true,
    });

    const events = buffer.split(/\r?\n\r?\n/);
    buffer = events.pop() ?? '';

    for (const eventText of events) {
      finished = parseSSEEvent(
        eventText,
        onDelta,
      );

      if (finished) break;
    }
  }

  if (!finished && buffer.trim()) {
    parseSSEEvent(buffer, onDelta);
  }
}

The most critical part here is:

buffer = events.pop() ?? '';

After splitting by blank lines, the last item of the array may be only half an event. It cannot be discarded and needs to be kept for the next round of concatenation.

Continuously update the page in Vue

Vue uses reactive state to save the question, answer, and loading status:

<script setup>
import { ref } from 'vue';

const question = ref('Write an English essay');
const content = ref('');
const loading = ref(false);

async function submit() {
  if (!question.value || loading.value) return;

  content.value = '';
  loading.value = true;

  try {
    const response = await fetch('/api/stream', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        prompt: question.value,
      }),
    });

    if (!response.ok) {
      throw new Error(`Request failed: ${response.status}`);
    }

    await readSSEStream(response, delta => {
      content.value += delta;
    });
  } catch (error) {
    content.value = error.message;
  } finally {
    loading.value = false;
  }
}
</script>

<template>
  <main class="container">
    <input v-model="question" />
    <button
      :disabled="loading"
      @click="submit"
    >
      {{ loading ? 'Generating...' : 'Submit' }}
    </button>

    <div class="output">{{ content }}</div>
  </main>
</template>

<style scoped>
.output {
  margin-top: 16px;
  white-space: pre-wrap;
}
</style>

Whenever a new delta is parsed:

content.value += delta;

Vue will update the corresponding page content, thus forming a typewriter-like output effect.

Text interpolation {{ content }} is used here instead of directly using v-html to render the model's answer, avoiding executing unprocessed model output as HTML.

Why do chat requests usually use Fetch instead of EventSource?

The browser's native EventSource can also receive SSE, but it is mainly oriented towards GET requests and is inconvenient for customizing request bodies and headers.

Large model chat usually requires:

Therefore, fetch() combined with ReadableStream is more suitable for chat scenarios.

How to choose between SSE and WebSocket?

Comparison Item SSE WebSocket
Communication Direction Server continuously sends to client Bidirectional communication
Protocol Basis HTTP WebSocket
Data Format Text events Text or binary
Chat Generation Suitable for server continuously returning answers Can also be implemented, but usually more complex

The core data direction for large model answer generation is:

Client submits a question once
Server continuously returns content

Therefore, SSE can already meet most text generation scenarios. Only when the business requires continuous bidirectional real-time communication is it necessary to further consider WebSocket.

What does the server need to provide?

The focus of this article is browser-side parsing, and the BFF implementation of /api/stream will be placed in the next article.

From the client's perspective, the server at least needs to:

Receive the frontend question
→ Carry the server-saved API Key to request the large model
→ Enable stream
→ Maintain streaming response
→ Continuously forward SSE data to the browser

Typical response headers include:

Content-Type: text/event-stream; charset=utf-8
Cache-Control: no-cache
Connection: keep-alive

The API Key is saved on the BFF server side and cannot be exposed to the browser using VITE_ environment variables.

Summary

The "typewriter effect" of large models is not simulated by a frontend timer, but a continuously working data pipeline:

LLM generates Token
→ Server encapsulates as SSE event
→ HTTP response stream
→ ReadableStream
→ Uint8Array
→ TextDecoder
→ buffer concatenates complete events
→ JSON.parse
→ Vue reactive update

There are three key boundaries:

After understanding these three boundaries, streaming output is no longer just "looping to read response.body," but a set of data processes that can stably handle network packet splitting, Chinese decoding, and incremental rendering.