AI Streaming Cuts Out at 90%: What 'Continue' Actually Does Under the Hood

Today I saw an interesting question and sent it to a friend:

AI generation suddenly stops at 90%: what's your solution?

My friend replied instantly: "Continue."

I said: "No tokens left, can't continue."

He replied with two more words: "Top up."

The almighty pay-to-win!

Jokes aside, if it's not a quota issue but the AI's streaming generation suddenly disconnects at 90%, does "continue" really work? Does it pick up from the breakpoint, or secretly start over from the beginning?

This seemingly simple operation can actually be three completely different things: resending content you didn't receive, continuing to write based on existing text, or restoring the model's computational state before the interruption.

Let's start with the conclusion:

"Recovery" in AI streaming generation includes at least three layers: connection recovery, content recovery, and inference recovery. The "continue generation" we see may not truly restore the model's computational scene before the interruption.

1. What happens when output stops at 90%

When ordinary text stops mid-sentence, users can usually still understand the preceding content. But AI output is increasingly consumed directly by programs: JSON renders forms, HTML generates pages, Markdown displays documents, and tool parameters might trigger real actions like sending emails or charging payments.

When this content breaks in the middle, the problem is more than just "missing a few words."

1.1 Plain Text: possible repetition, missing words, or style shifts

Plain text is the most fault-tolerant. Even if it stops mid-sentence, the part already received is usually still readable.

But if the original model request has already failed, the system can only re-request the model to continue writing. New content might repeat the previous paragraph, or suddenly change tone, person, or even conclusion. Because "re-continuing" does not equal "restoring the scene"; it's just another inference based on existing text.

1.2 JSON: missing one bracket makes the entire object unusable

The model might stop here:

{
  "title": "AI Interruption Recovery",
  "items": [
    {"name": "Event Replay"},
    {"name": "KV Cache"

At this point, a closing quote, closing brace, and array terminator are missing. Calling JSON.parse() directly will definitely throw an error. If the recovered model starts outputting from a new {, simply concatenating the two segments is useless.

A safer approach is:

• Save the raw string during the streaming phase; don't treat each chunk as complete JSON;
• Use an incremental parser that preserves the parsing state of quotes, escape characters, arrays, and objects;
• Or break structured output into independent events like field_update or JSON Patch;
• After receiving completed, perform a full schema validation;
• When regenerating, create a new revision; don't directly concatenate results from two generations.

1.3 HTML: displayable doesn't mean structurally correct

The model might only generate up to:

<section class="card">
  <h2>Recovery Plan</h2>
  <p>First, save the event log

Browsers typically try to auto-complete missing tags, so the page might not immediately go blank. However, layout shifts, node nesting errors, or stuffing all subsequent content into the same element can occur.

If the model-generated HTML is directly written into innerHTML or a framework's v-html, it also introduces XSS risks. Streaming interruption and recovery cannot bypass existing security policies.

In practice, you should:

• Prioritize plain text or code preview during the streaming phase;
• If HTML preview is necessary, strictly sanitize it first, then place it in a restricted sandbox iframe;
• Do not execute scripts, event attributes, or external resources during the half-finished stage;
• After receiving the completion event, perform a full parse, sanitize, and render.

1.4 Markdown: no errors, but display can be completely messed up

Common interruption points in Markdown include:

• Code fences not closed, causing all subsequent text to be displayed as code;
• Tables only half-generated, with column counts constantly changing;
• Links only generated as [title](, causing subsequent content to render abnormally;
• Bold, lists, or blockquotes stopping mid-way, causing continuous page jitter.

The client can use a Markdown renderer that supports incremental updates, or only refresh at complete paragraph, code block, or event boundaries. Fences or tags temporarily added for preview purposes must only stay in the display layer and cannot be written back to the real content.

1.5 Unicode: a single Chinese character can be split in half

Network fragments (chunks) are transmitted in bytes and don't guarantee landing exactly on character boundaries. UTF-8 Chinese characters, Emojis, or combining characters can all span two fragments. If each fragment is decoded individually, the client might display `` or garbled text.

The receiving end needs a decoding method that supports streaming state. For example, a web client can call TextDecoder.decode(chunk, { stream: true }), allowing the decoder to temporarily store incomplete byte sequences.

1.6 Code and Agents: the danger isn't just display anomalies

When code is interrupted, functions, strings, and comments might all be unclosed. Automatically executing tests, deploying, or directly overwriting original files at this point can amplify the failure. A safer approach is to first mark it as draft, complete a syntax check, then write to disk via temporary files, patches, or atomic replacement.

The risk of Agent tool calls is even higher. Tool parameters often arrive as JSON increments and might only be half-received during an interruption. If directly retried after recovery, it could also result in duplicate emails, duplicate charges, or duplicate record creation.

Therefore, tool calls at least require:

• Aggregating parameters for the same call via tool_call_id;
• Executing only after receiving the completion marker and passing schema validation;
• Setting idempotency keys for operations with side effects;
• Saving pending / running / succeeded / failed status;
• When recovering, first query whether the original operation has already succeeded before deciding to retry.

These seemingly different problems can actually be summarized by the same principle:

Half-finished products can be displayed, but cannot be parsed, executed, or submitted as complete data. Raw increments, parsing state, and business submission state must be saved separately.

2. A disconnected page doesn't mean the generation task has ended

To understand how to recover, we must first find where the failure occurred. A typical AI streaming generation roughly passes through this chain:

Model Service → Streaming Request → Business Service → SSE / WebSocket / Streaming HTTP → Client

So-called "generation suddenly stops at 90%" can happen at any segment.

Type 1: Client connection dropped, but the model is still generating

This is the most common and easiest case to handle.

The user refreshed the web page, the mobile device switched networks, the desktop app entered sleep mode, or the streaming connection was closed by a gateway, but the background task and model request haven't stopped. The model continues generating, and the server continues saving results.

After the client reopens the session, the server just needs to resend the parts it missed. This scenario allows for precise recovery because the missing content has actually already been generated; it just wasn't successfully delivered to the current client.

Type 2: Connection between business service and model dropped

This is where trouble gets serious.

The model might have only generated up to a certain token before the upstream streaming request failed. If the model interface cannot continue the original task, the business service usually cannot obtain the model's internal computational scene at that moment.

The system can only retain the received text, put it back into the Prompt, ask the model to "continue from here, don't repeat," and then initiate a new model request.

This is semantic continuation, not precise continuation.

Type 3: The generation service itself crashed

If task status, generated text, and progress only exist in process memory, then once the process crashes, you basically have to start over.

More mature systems assign generation tasks to independent Workers and continuously record task status and output events. This way, even if a service instance restarts, it knows where the task was, what content has been generated, and whether a retry is needed.

So before discussing recovery, we need to answer:

Was it the client subscription connection, the model request, or the entire generation task that dropped?

3. "Continue" can be three different operations

Only after finding the breakpoint can we decide how to "continue." The product interface has just one interaction, but internally it might execute three completely different operations.

3.1 Replay

The server has already generated the content, but the client didn't receive it.

The system just needs to resend the missing events; there's no need to call the model again. The content can remain completely consistent. This is the most reliable and lowest-cost recovery method.

3.2 Re-continuation

The original model request has failed. The system re-submits the existing text to the model and asks it to continue completing it.

It will produce a new inference, potentially repeating, omitting, or changing style. For JSON, HTML, code, and tool calls, structural breaks and duplicate side effects must also be handled.

3.3 Inference State Recovery

The system retains or reloads the KV Cache, token sequence, and sampling state, allowing the model to continue generating from near the original computational scene.

This is closest to truly "continuing from the model's breakpoint," but implementation is complex and typically requires a self-built inference service.

Operation	What actually happens	Can it recover precisely?
Replay	Resends already generated events	Yes
Re-continuation	Initiates a new model inference	Not necessarily
Inference State Recovery	Loads the inference state of the original task	Depends on state completeness

4. How do real projects typically handle client disconnections?

A common pitfall in real projects is: as soon as the client connection drops, the model request is immediately cancelled.

This misinterprets page refreshes, network switches, app sleep, and gateway timeouts as the user actively giving up.

A safer approach is to decouple the "generation task" from the "client connection":

The client is just a subscriber to the results, not the owner of the generation task.

After disconnection, the background task can continue running for a period, writing new content to the event log. When the client returns, it resumes subscribing from the last received position.

Create generation task
    ↓
Model continuously outputs
    ↓
Server records events
    ↓
Push to client
    ↓
Client disconnects
    ↓
Task continues running or enters grace period
    ↓
Client reconnects with last_seq

Different tasks can adopt different strategies:

• Regular chat: continue generating and cache after disconnection;
• Long reports or code: set a 30-60 second grace period, decide to cancel after timeout;
• Background tasks like images, videos: continue execution, independent of client connection;
• Real-time voice: stop as soon as possible after disconnection to avoid continuous resource consumption.

The system must also distinguish between "network disconnection" and "user actively stopping." When the user clicks stop, the client should call an independent cancel interface, not simply close the SSE or WebSocket.

5. How does the event log ensure no content duplication or loss?

Each output fragment can be recorded as an event with an incrementing sequence number:

{
  "generation_id": "gen_123",
  "revision": 1,
  "seq": 58,
  "type": "text_delta",
  "text": "This is the newly generated content"
}

Here, generation_id identifies the generation task, revision distinguishes versions after regeneration, and seq is responsible for ordering, gap detection, and deduplication. If the client has continuously received up to seq=57, upon reconnection it asks the server to resend starting from 58.

Client already has: 1 ~ 57
Server resends: 58 ~ 93
Subsequent new events: 94, 95, 96...

The client deduplicates by generation_id + revision + seq, so even if the server sends duplicates, there won't be duplicate concatenation. Recent events can be placed in short-term storage like Redis Streams, with final results written to the database; long tasks use "text snapshot + small increments" to avoid replaying from the first token.

Structured content must also additionally save parsing state: validate JSON after completion, sanitize HTML before rendering, and execute tool calls only after parameters are complete and idempotency checks pass.

6. Can KV Cache allow the model to "pick up right where it left off"?

Event logs solve the problem of content resending. But if the model request really stops, regenerating might still require recomputing all preceding text.

KV Cache saves the Keys and Values generated during attention computation for historical tokens. Think of it as the model's "calculation draft":

Text: What the model has already read and generated
KV Cache: The intermediate calculation results produced when the model processed these tokens

If the model has already processed 10,000 tokens, without KV Cache it must re-execute the prefill; if the inference service retains a reusable KV Cache, it can skip most of the preceding text computation and generate the next token faster.

Event logs solve "don't let the user lose content," KV Cache solves "don't let the model recompute the preceding text."

However, having KV Cache doesn't automatically grant "breakpoint resume." It cannot replace event logs, and its size is very large. Cross-node recovery might also involve:

GPU → CPU → Network Storage → CPU → Another GPU

During recovery, compatibility of model version, Tokenizer, cache format, and sampling state must also be ensured. Third-party model APIs typically don't hand KV Cache over to the application for management, so the vast majority of products still primarily use event replay; only when the original task genuinely fails do they downgrade to semantic continuation.

Final Thoughts

Reliable AI products need to manage connection, content, and inference states separately: event logs ensure content is replayable, task systems decouple generation from any single client connection, and KV Cache reduces redundant computation in worthwhile scenarios.