The Five-Layer Pipeline That Turns LLM Tokens Into Streaming Text
Every LLM API that supports streaming uses the same SSE-over-HTTP pattern, but the edge cases—truncated JSON across packet boundaries, the `[DONE]` sentinel, multi-byte character splitting—are what turn a demo into something that doesn't silently drop characters. Understanding the buffer-and-retry mechanism means you can implement streaming for any provider without relying on a library that hides these failure modes.
Streaming output from an LLM is not a single operation but a five-layer pipeline that transforms raw network bytes into characters on a screen. Each call to `reader.read()` pulls a chunk of binary data that may contain zero, one, or multiple lines of SSE-formatted text, and a single JSON line can be split across two network packets. A `buffer` variable catches these truncated fragments so they can be stitched together on the next iteration instead of being discarded.
The pipeline runs inside a `while` loop: decode the binary chunk, prepend any buffered residue, split on newlines, filter for lines starting with `data:`, strip that prefix with `slice(6)`, check for the `[DONE]` sentinel, and parse the remaining JSON to extract `delta.content`. A `try-catch` block catches incomplete JSON and stores it back into the buffer with the `data:` prefix restored, ensuring the format stays consistent for the next round.
Two assignment strategies separate streaming from non-streaming modes. Streaming uses `content.value += delta` to append each token incrementally, while non-streaming calls `response.json()` once and assigns the full `message.content` with `=`. The `[DONE]` flag is a plain-text signal embedded in the SSE stream, not a TCP-level close, so the code must manually set `done = true` to exit the while loop.
The entire streaming implementation hinges on a single design decision: never discard data on error. The try-catch block is not for error recovery but for deferred assembly, treating a parse failure as a signal that the message is incomplete rather than malformed.
The buffer variable is empty in the common case and only populated when a truncation occurs, which means the happy path pays no overhead for the safety net.
Restoring the `data:` prefix inside the catch block is a subtle detail that, if missed, causes silent data loss—the fragment won't pass the filter on the next iteration and disappears without an error.
The [DONE] check and the reader.read() done flag are two independent termination conditions that must both be handled because the server can signal completion either through the stream or through an in-band message.