Python · Backend · LLM

SSE Turns Python Generators into Streaming LLM Chat Replies

By Dragon_xjy · Nov 2, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Chat interfaces get instant feedback instead of a frozen wait; a single generator wrapped in StreamingResponse turns any async LLM stream into an SSE stream the browser can render incrementally.

Summary

Streaming a large model's response to a web page relies on three pieces: a generator that yields partial chunks, an SSE response wrapper, and a frontend that reads frames and appends them to the DOM. In FastAPI, `yield` suspends the function after returning each piece, and `StreamingResponse` keeps reading from that generator and sending each yielded value over the wire. The response must use `media_type="text/event-stream"`, and headers like `Cache-Control: no-cache` and `X-Accel-Buffering: no` prevent proxies from delaying the stream.

The backend reads from an existing `agent.execute_stream(messages)` method with `async for`, handling two event types: `reasoning` fragments go to a thinking area, and `answer` fragments accumulate into the final reply. Each event is JSON-serialized and wrapped in an SSE frame with `event:` and `data:` lines, terminated by a blank line. A final `done` event signals completion; an `error` event carries a user-readable failure message.

On the client, plain `fetch` reads the response body with `getReader()`, decodes with `TextDecoder` (preserving partial multi-byte characters), splits frames on blank lines, and appends content based on the JSON `type` field. An `AbortController` lets the user stop the stream mid-generation, canceling the reader and restoring the send button. The whole pattern works for any generator-based agent, not just this one.

Takeaways

— `yield` in a Python function suspends execution and returns a chunk, resuming from that point on the next iteration.

— An SSE message is formatted as `event: type\ndata: content\n\n`; the trailing blank line marks the end of a message and does not close the connection.

— FastAPI's `StreamingResponse` takes a generator and sends each value with `media_type="text/event-stream"`.

— Setting `Cache-Control: no-cache` forces revalidation, and `X-Accel-Buffering: no` disables Nginx buffering so chunks forward immediately (when the proxy allows it).

— The backend serializes each event with `json.dumps(..., ensure_ascii=False)` so newlines inside content don't break SSE frame boundaries.

— The frontend uses `response.body.getReader()` with `TextDecoder` to stream-decode; `decoder.decode(value, {stream: !done})` preserves incomplete UTF-8 sequences.

— Frames are split on `\r?\n\r?\n`; incomplete frames remain in a buffer for the next read.

— An `AbortController` stops the request mid-stream; the reader is cancelled and the send button re-enabled in `finally`.

— The agent stream returns `{"type":"reasoning","content":...}` and `{"type":"answer","content":...}` chunks; the frontend appends them to separate areas based on `type`.

Conclusions

The pattern keeps the generator as the single source of truth: the backend never buffers the full reply, and the frontend never reassembles JSON events, because the abstraction already separates reasoning from answer.

Handling incomplete UTF-8 characters is a real-world detail that only surfaces with Chinese text; the `stream: !done` flag shows why streaming decoders are not optional for non-Latin scripts.

The `X-Accel-Buffering` header is a reminder that a working SSE setup depends on the proxy in front of the app, not just the framework; forgetting it leaves the user staring at a blank page until the stream flushes.

Concepts & terms

SSE (Server-Sent Events)

A one-way HTTP streaming protocol where the server pushes text frames over a long-lived response. Each frame is a set of `field: value` lines separated by blank lines; the browser's EventSource API can consume it, but a raw fetch with a reader also works.

StreamingResponse

FastAPI's response class that accepts a generator and sends each yielded chunk immediately, with configurable media type and headers. It keeps the connection open and flushes data as the generator produces it.

X-Accel-Buffering

An Nginx response header that tells the proxy to disable buffering for a specific response. When set to `no`, Nginx forwards chunks to the client as soon as they arrive, which is required for SSE to feel live; the proxy must be configured to allow the header.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗