← Back to the summary

Streaming RAG Isn't a Flag — It's a Full-Stack Latency Problem

Some time ago, a friend interviewed for an algorithm position at Tencent. His resume stated, "Optimized average RAG system response time to within 3 seconds."

The interviewer glanced at it and asked:

"Is that 3 seconds end-to-end latency or Time To First Token?"

He paused: "End-to-end... it takes 3 seconds for the user to get the complete answer."

The interviewer followed up:

"So how long does the user wait from sending the question to seeing the first character? Have you implemented streaming output? What's the difference between streaming RAG and regular streaming output? What unique latency issues need to be handled in RAG scenarios? Can the retrieval phase be streamed, or is only the generation phase streamable?"

Four questions, and he could only answer the first one—because his system only did streaming output for the generation phase, without any consideration for latency optimization in the retrieval phase.

The interviewer remarked: "End-to-end 3 seconds, but if the first 2 seconds are spent waiting for retrieval, the user experience is still poor. Streaming output addresses perceived latency, not actual latency."

This scenario highlights a problem: many people implement streaming output but haven't systematically considered the sources of latency and optimization strategies in RAG scenarios.

Today, let's break down streaming RAG from the ground up.

First, Understand Two Latency Concepts

Before optimizing latency, you must distinguish between two concepts:

TTFT (Time To First Token): First Token Latency

The time from when the user sends a request to when the first output character appears.

This is the most direct user perception metric—even if the answer takes 10 seconds to generate fully, as long as the first character appears within 1 second, the user's waiting anxiety drops dramatically.

End-to-End Latency (E2E Latency): Complete Response Time

The total time from the request being sent to the complete answer being returned.

This is the actual time consumed by the system, determining throughput and cost.

Why TTFT is High in RAG Scenarios:

Regular LLM call: Request sent → LLM starts generating → Streams the first token. TTFT is just the LLM's inference startup time, typically 500ms~1 second.

RAG call: Request sent → Query rewriting (may call LLM) → Vector retrieval → Rerank → Assemble prompt → LLM starts generating → Streams the first token. Before the LLM even starts generating, there's already a 2~~5 second retrieval pipeline delay, pushing TTFT up to 3~~6 seconds.

The user waits 3 seconds to see the first character. Even if the subsequent streaming output is smooth, the experience is poor.

The Core Idea of Streaming RAG

The goal of streaming RAG is: Let the user see the first character as early as possible, without compromising answer quality.

There are two directions to work on:

Direction 1: Optimize the retrieval pipeline to shorten the waiting time before TTFT.

Make retrieval faster so the LLM can start generating sooner.

Direction 2: Stream intermediate processes to give users feedback while they wait.

The retrieval phase is unavoidable, but you can push progress information like "Retrieving..." to the user in real-time to reduce waiting anxiety.

Both directions must be pursued simultaneously; neither is dispensable.

Pitfall 1: Can Query Rewriting Be Made Asynchronous?

In the standard RAG pipeline, Query Rewriting is the first step—rephrasing the user's question into a form more suitable for retrieval.

This step usually requires calling an LLM, taking 500ms~1 second.

Question: Must query rewriting be completed synchronously before retrieval?

Not necessarily. There are two optimization approaches:

Approach 1: Determine if rewriting is needed; skip it if not.

If the user's question is a simple single-hop question, has no anaphora, and is semantically clear, directly use the original question for retrieval, skipping the rewriting step.

You can use rules to quickly judge (only trigger rewriting if it contains pronouns like "it," "this," "that"; otherwise, retrieve directly).

This way, 70%~~80% of queries can skip rewriting and go straight to retrieval, reducing TTFT by 500ms~~1 second.

Approach 2: Run rewriting and retrieval in parallel.

Initiate two paths simultaneously: retrieve with the rewritten question + retrieve with the original question, run them in parallel, and take the path with better results.

Before rewriting is complete, the retrieval with the original question might already have results. If the result quality is good enough, use it directly; if the rewritten results are better, switch to them.

This way, TTFT is not dragged down by rewriting latency. Rewriting becomes an "optional quality improvement" rather than a "must-wait step."

Practical issue with Approach 2: How to judge "better results"?

After the two parallel paths return, a quality judgment logic is needed to decide which path to use. Typically, compare the Top-1 similarity scores. Only replace if the rewritten Top-1 score exceeds the original by a certain threshold (e.g., 0.05); otherwise, use the original results directly.

How to set the threshold? There's no universal value. You need to label a batch of "rewriting effective/ineffective" samples on your own dataset and observe the distribution of score differences to determine it. In our project, we finally settled on 0.03—too small, and rewriting almost always replaces (losing the latency-saving benefit); too large, and the benefit of rewriting doesn't materialize.

Pitfall 2: Parallelization in the Retrieval Phase

The standard RAG retrieval pipeline is usually serial:

Vector Retrieval → BM25 Retrieval → RRF Fusion → Rerank → Send to LLM

Which steps can be parallelized:

Vector retrieval and BM25 retrieval have no dependencies on each other and can be fully initiated in parallel. Perform RRF fusion after both paths return results.

If there are multiple vector database shards (common in large knowledge bases), shard queries can be parallelized.

After parallelization, how to determine the waiting strategy for RRF fusion?

Here's a practical trade-off: wait for both paths to complete before fusing (safe, but the slower path drags down the whole process), or set a timeout and discard the path that times out?

In production, the latter is usually chosen: set a soft timeout (e.g., 800ms). Any retrieval path that hasn't returned within this time is simply abandoned, and fusion is done with the available results. This way, the worst-case retrieval latency has an upper bound, preventing a single path's jitter from dragging down the overall TTFT.

Beyond the soft timeout, you also need to pay attention to concurrency pressure—two parallel paths mean the vector database and ES receive requests simultaneously, doubling the peak concurrency. If downstream services aren't properly rate-limited, parallelization might actually trigger degradation, making latency longer than serial execution. We added semaphore control on the vector database side, allowing a single request to initiate at most 3 parallel retrieval paths; any excess wait in a queue.

Can Rerank be parallelized:

Rerank is a fine-ranking of recalled results. It must wait for both vector retrieval and BM25 to complete before starting, so it cannot be parallelized.

However, the truncation point for Rerank can be dynamically adjusted—if the retrieval result quality is already very high (Top-1 score exceeds a threshold), you can skip Rerank and use the retrieval results directly, saving Rerank's time (typically 200~500ms).

Here, too, there's a threshold calibration problem. The absolute value of vector similarity has completely different meanings under different embedding models—a score of 0.85 represents a very different semantic distance with ada-002 versus bge-large. The threshold for skipping Rerank must be calibrated on your own embedding model and knowledge base; you cannot directly use numbers from someone else's project. Calibration method: compare the answer quality of "skip Rerank" versus "full Rerank" on a test set, and find the minimum score threshold that doesn't affect quality.

Measured Data:

In our insurance knowledge base project, after parallelizing vector retrieval and BM25, the retrieval phase latency dropped from 800ms to 450ms, a reduction of about 44%. After adding soft timeout protection and concurrency limiting, P99 latency dropped from the original 2.1 seconds to 1.3 seconds, with long-tail jitter significantly converging.

Pitfall 3: Streaming Output in the Generation Phase, RAG-Specific Issues

Streaming output in the generation phase seems simple—the LLM supports a streaming API, just use it directly.

But there's a specific problem in RAG scenarios: Citation annotation.

In standard RAG, after the answer is generated, the system appends citations at the end ("[Source: Product Manual A - Page 3]").

With streaming output, the LLM generates and outputs tokens as it goes. During generation, it doesn't yet know where citation annotations should be placed or which sources should be cited.

Solution 1: Append citations after the streaming output finishes.

The LLM streams the answer body; after generation is complete, the system uniformly appends the citations.

Simple to implement, but the user has to wait for the complete answer before seeing the sources, which isn't elegant.

Solution 2: Instruct the LLM in the prompt to annotate inline during generation.

In your answer, whenever you reference information from the provided materials, immediately annotate it after that sentence with [Source N],
e.g., The waiting period for Critical Illness Insurance A is 90 days [Source 1], with a deductible of 5000 yuan [Source 2].

The LLM annotates inline during generation, and citations appear alongside the answer text during streaming output.

Slightly more complex to implement (need to maintain the mapping between source numbers and chunks in the prompt), but the user experience is better.

Production issue with Solution 2: LLM's citation annotations are not trustworthy

Here's a pitfall not mentioned in the article but must be handled in production: LLMs sometimes hallucinate source numbers. You pass in 3 chunks, and it might write [Source 4], [Source 5], or format them as [Source 1], [Source1], (Source 1), in a wide variety of formats.

Post-processing for streaming output needs to specifically handle this:

Validity Check: After streaming output finishes, use regex to extract all [Source N], filtering out hallucinated citations where N exceeds the number of chunks;
Format Normalization: Use regex to unify various format variants into a standard format before rendering to the user;
Fallback Plan: If the citation format is detected to be chaotic (e.g., multiple formats appear in the same paragraph), downgrade to Solution 1, uniformly appending the source list at the end.

This logic isn't complicated to write, but if you don't write it, it will definitely fail in production—the problems QA finds are basically these format issues.

Pitfall 4: Streaming Progress Feedback, User Experience in the Retrieval Phase

The retrieval phase cannot be truly streamed (generation must wait for retrieval to complete), but progress information can be pushed to the user in real-time.

Implementation of Progress Feedback: SSE (Server-Sent Events)

The server pushes progress events to the client in real-time via an SSE long connection:

event: progress
data: {"stage": "query_rewrite", "message": "Understanding your question...", "progress": 10}

event: progress  
data: {"stage": "retrieval", "message": "Retrieving relevant documents...", "progress": 40}

event: progress
data: {"stage": "rerank", "message": "Filtering the most relevant content...", "progress": 70}

event: token
data: {"token": "Acc", "cumulative": "Acc"}

event: token
data: {"token": "ording", "cumulative": "According"}

The user sees:

[Understanding your question... 10%]
[Retrieving relevant documents... 40%]  
[Filtering the most relevant content... 70%]
According to the Critical Illness Insurance A product manual, the waiting period is... (streaming output)

Psychological research shows that with progress feedback, the acceptable waiting time for users is 2~3 times longer than without it. Even if retrieval still takes 3 seconds, seeing the progress bar moving makes the waiting experience completely different.

Several easy pitfalls with SSE in production:

Pitfall 1: Nginx's default buffering breaks SSE.

Nginx buffers upstream responses by default, forwarding to the client only when the buffer is full. SSE requires real-time, event-by-event pushing. Buffering turns it into batch sending, completely eliminating the progress bar effect.

The solution is to disable buffering for the corresponding route in the Nginx configuration:

location /api/sse {
    proxy_pass http://backend;
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header X-Accel-Buffering no;
}

Missing this config means it works fine in local testing but breaks upon deployment—because local testing doesn't go through Nginx.

Pitfall 2: Event deduplication on client reconnection.

The SSE protocol has a built-in reconnection mechanism. After disconnection, the client automatically reconnects and carries the Last-Event-ID header, telling the server from which event to start resending.

If the server doesn't handle this header, it will push from the beginning again after reconnection. The user sees the progress bar reset to zero and start over, or receives duplicate tokens.

The solution is to add an id field to each event. The server records the maximum event_id sent and resumes sending from events after that id upon reconnection:

id: 42
event: token
data: {"token": "ding"}

Pitfall 3: Connection stickiness under load balancing.

SSE is a long connection. All events for the same session must go to the same backend instance (because the event sequence state is in memory).

If the load balancer uses round-robin per request, a reconnection after disconnection might be routed to a different instance, which won't find the original session state.

The solution is to configure sticky sessions based on session ID, or store the event sequence state in Redis so any instance can resume sending.

Pitfall 5: Error Handling for Streaming Output

If an error occurs mid-stream after streaming output has started (network interruption, LLM timeout, empty retrieval results), handling is more complex than for regular requests.

Error Scenario 1: Error during the retrieval phase.

Streaming output hasn't started yet. Handle it like a regular error, returning an error message to the user.

Error Scenario 2: Error mid-generation.

Streaming output has already started, the user has seen part of the answer, and then the LLM errors or times out.

You can't just return an error, because the user has already seen partial content. An abrupt interruption is a terrible experience.

Handling: Append "[Answer generation interrupted, please retry]" after the already-output content, and provide a "Continue Generation" button on the frontend to trigger a retry.

Note that "resuming from the interruption point" is practically infeasible in production—LLM generation is non-deterministic, and a retry will almost certainly not follow the original path. Forced concatenation creates semantic breaks. The actual practice is to regenerate from scratch, but pass the already-output partial content as context to the LLM to maintain as much consistency as possible, rather than true breakpoint continuation.

Error Scenario 3: Problematic content detected in the streaming output.

Content safety checks are usually done on the complete answer. With streaming output, the answer is still being generated—how to handle this?

Plan A: Buffer the complete answer for safety checks first, then stream it after passing (losing the TTFT advantage of streaming).

Plan B: Use lightweight real-time content filtering (keyword filtering), detecting during generation, immediately interrupting and appending a prompt if triggered.

These two plans are essentially a trade-off between safety and experience, with no universal answer. Risk-sensitive businesses (finance, healthcare) usually choose Plan A, preferring to sacrifice some TTFT rather than let problematic content reach the user before checks complete. Scenarios with lower content risk can choose Plan B, using lightweight filtering as a fallback, with full safety checks running asynchronously and intervening in subsequent sessions if issues are found.

Prioritization of End-to-End Latency Optimizations

Combining all the optimization points discussed above, ranked by benefit/cost ratio:

First Priority: Streaming output for the generation phase

Low implementation cost, huge TTFT benefit. Almost all systems should do this.

Second Priority: Retrieval parallelization (Vector + BM25 in parallel)

Medium implementation cost, reduces retrieval phase latency by 40%+. Significant benefit. Remember to add soft timeout protection and concurrency limiting simultaneously, otherwise peak traffic might backfire.

Third Priority: Streaming progress feedback

Medium implementation cost. Doesn't change actual latency but significantly improves user experience perception. Worth doing. Remember to handle the three pitfalls: Nginx buffering, disconnection reconnection, and load balancing. Otherwise, it works in the test environment but fails in production.

Fourth Priority: Conditional query rewriting

Low implementation cost. Saves 500ms~1 second for simple queries, no impact on complex queries.

Fifth Priority: Dynamically skipping Rerank

Low implementation cost. Saves 200~500ms for high-quality retrieval result scenarios, but the threshold must be calibrated on your own dataset; you cannot borrow numbers from others.

How to Answer in an Interview

When the interviewer asks "How did you implement streaming output?" or "How do you optimize user waiting experience?", expand like this:

First, distinguish between TTFT and end-to-end latency. Clearly explain the difference between the two. The reason TTFT is high in RAG scenarios is the retrieval pipeline before LLM generation. This demonstrates a systematic understanding of latency issues.

Then, talk about retrieval phase optimizations. Parallelizing vector retrieval and BM25, the waiting strategy for parallel execution (soft timeout + concurrency limiting), conditional query rewriting, dynamically skipping Rerank. Mention specific performance data.

Next, discuss streaming in the generation phase. The RAG-specific problem—how to handle citation annotations, the pros and cons of the two solutions, and the production pitfalls of Solution 2 (post-processing for hallucinated citations and format instability).

Then, talk about progress feedback. The implementation method using SSE, focusing on the three production details: Nginx buffering, disconnection reconnection, and load balancing. Mentioning these scores significant bonus points.

Finally, discuss error handling. Why streaming breakpoint continuation is infeasible in production, the two plans for content safety checks and their applicable scenarios. This shows you've considered real production constraints.

One Last Thing

Many people think "adding a stream=True parameter" counts as implementing streaming output for streaming RAG.

In reality, latency optimization in RAG scenarios is a systemic problem—retrieval parallelization, conditional query rewriting, progress feedback, error handling. Every link has room for optimization and its own pitfalls.

But more importantly, when articulating these, you need to clearly explain the trade-offs behind each decision: why the soft timeout threshold is set that way, how the score threshold for skipping Rerank is calibrated, why Nginx buffering must be turned off for SSE. Explaining the trade-offs lets the interviewer know you've actually done it, not just recited conclusions from an article.