Streaming RAG Isn't a Flag: The Five Latency Traps That Kill User Experience
Some time ago, a friend interviewed for an algorithm position at Tencent. His resume stated, "Optimized average RAG system response time to under 3 seconds."
The interviewer glanced at it and asked:
"Is that 3 seconds end-to-end latency or time-to-first-token?"
He paused: "End-to-end... it takes 3 seconds for the user to get the complete answer."
The interviewer followed up:
"So how long does the user wait from sending the query to seeing the first character? Have you implemented streaming output? What's the difference between streaming RAG and regular streaming output, and what unique latency issues need to be handled in RAG scenarios? Can the retrieval phase be streamed, or is streaming only possible during the generation phase?"
He could only answer the first of those four questions—because his system only had streaming output for the generation phase and hadn't considered latency optimization for the retrieval phase at all.
The interviewer remarked: "End-to-end 3 seconds, but if the first 2 seconds are spent waiting for retrieval, the user experience is still poor. Streaming output addresses perceived latency, not actual latency."
This scenario highlights a problem: many people implement streaming output without systematically thinking about the sources of latency and optimization strategies in RAG scenarios.
Today, let's break down streaming RAG from the ground up.
First, Clarify Two Latency Concepts
Before optimizing latency, you must distinguish between two concepts:
TTFT (Time To First Token): First-token latency
The time from when the user sends a request to when the first output character appears.
This is the most direct metric for user perception—even if the answer takes 10 seconds to fully generate, as long as the first character appears within 1 second, the user's waiting anxiety drops significantly.
End-to-End Latency (E2E Latency): Complete response time
The total time from when the request is sent to when the complete answer is returned.
This is the actual time consumed by the system, determining throughput and cost.
Why TTFT is high in RAG scenarios:
A regular LLM call: Request sent → LLM starts generating → Streams the first token. TTFT is just the LLM's inference startup time, typically 500ms~1 second.
A RAG call: Request sent → Query rewriting (may call an LLM) → Vector retrieval → Rerank → Assemble prompt → LLM starts generating → Streams the first token. Before the LLM even begins generating, there's already a 25 second retrieval pipeline latency, pushing TTFT up to 36 seconds.
The user waits 3 seconds to see the first character. Even if the subsequent streaming output is smooth, the experience is poor.
The Core Idea of Streaming RAG
The goal of streaming RAG is: Let the user see the first character as early as possible, without degrading answer quality.
There are two directions to pursue:
Direction 1: Optimize the retrieval pipeline to shorten the waiting time before TTFT
Make retrieval faster so the LLM can start generating sooner.
Direction 2: Stream intermediate processes to give the user feedback while waiting
The retrieval phase can't be avoided, but progress information like "Retrieving..." can be pushed to the user in real-time to reduce waiting anxiety.
Both directions must be pursued simultaneously; neither is dispensable.
Pitfall 1: Can Query Rewriting Be Made Asynchronous?
In a standard RAG pipeline, query rewriting is the first step—rephrasing the user's question into a form more suitable for retrieval.
This step usually requires calling an LLM, taking 500ms~1 second.
Problem: Must query rewriting be completed synchronously before retrieval?
Not necessarily. There are two optimization approaches:
Approach 1: Determine if rewriting is needed; skip it if not.
If the user's question is a simple single-hop question, contains no anaphora, and is semantically clear, use the original question directly for retrieval and skip the rewriting step.
A rule can quickly decide (trigger rewriting only if the question contains pronouns like "it," "this," "that"; otherwise, retrieve directly).
This way, 70%80% of queries can skip rewriting and go straight to retrieval, reducing TTFT by 500ms1 second.
Approach 2: Run rewriting and retrieval in parallel.
Initiate two paths simultaneously: retrieve with the rewritten question + retrieve with the original question, both in parallel, and take the path with better results.
Before rewriting completes, the original question's retrieval might already have results. If the result quality is good enough, use it directly; if the rewritten results are better, swap them in.
This way, TTFT isn't dragged down by rewriting latency. Rewriting becomes an "optional quality boost" rather than a "must-wait step."
Practical problem with Approach 2: How to judge "better results"?
After both parallel paths return, a quality judgment logic is needed to decide which path to use. Typically, compare the Top-1 similarity scores: only replace if the rewritten Top-1 score exceeds the original by a certain threshold (e.g., 0.05); otherwise, use the original results directly.
How to set the threshold? There's no universal value. You need to label a batch of "rewriting effective/ineffective" samples on your own dataset and observe the distribution of score differences to determine it. In our project, we settled on 0.03—too small, and rewriting almost always replaces (losing the latency-saving benefit); too large, and the benefit of rewriting doesn't materialize.
Pitfall 2: Parallelization in the Retrieval Phase
The retrieval pipeline in standard RAG is usually serial:
Vector retrieval → BM25 retrieval → RRF fusion → Rerank → Send to LLM
Which steps can be parallelized:
Vector retrieval and BM25 retrieval have no dependency on each other and can be initiated completely in parallel. Perform RRF fusion after both paths return results.
If there are multiple vector database shards (common with large knowledge bases), shard queries can be parallelized.
After parallelization, how to decide the waiting strategy for RRF fusion?
Here's a practical trade-off: wait for both paths to complete before fusing (safe, but the slower path holds up the whole process), or set a timeout and discard any path that exceeds it?
In production, the latter is usually chosen: set a soft timeout (e.g., 800ms). Any retrieval path that hasn't returned within this time is simply abandoned, and fusion proceeds with the available results. This way, the worst-case retrieval latency has an upper bound, preventing a single jittery path from dragging down the overall TTFT.
Beyond the soft timeout, concurrency pressure also needs attention—two parallel paths mean the vector database and ES receive requests simultaneously, doubling peak concurrency. If downstream isn't properly rate-limited, parallelization might trigger degradation, making latency worse than serial execution. We added semaphore control on the vector database side, allowing a single request to initiate at most 3 parallel retrieval paths; any excess queues up and waits.
Can Rerank be parallelized:
Rerank is a fine-ranking of recalled results. It must wait for both vector retrieval and BM25 to complete before starting, so it cannot be parallelized.
However, the truncation point for Rerank can be dynamically adjusted—if the retrieval result quality is already very high (Top-1 score exceeds a threshold), Rerank can be skipped entirely, saving its latency (typically 200~500ms).
Again, there's a threshold calibration problem here. The absolute value of vector similarity has completely different meanings under different embedding models—a score of 0.85 from ada-002 and from bge-large represents very different semantic distances. The threshold for skipping Rerank must be calibrated on your own embedding model and knowledge base; you can't directly use numbers from someone else's project. Calibration method: compare answer quality between "skip Rerank" and "full Rerank" on a test set, and find the minimum score that doesn't affect quality.
Measured data:
In our insurance knowledge base project, after parallelizing vector retrieval and BM25, the retrieval phase latency dropped from 800ms to 450ms, a reduction of about 44%. After adding soft timeout protection and concurrency rate limiting, P99 latency dropped from the original 2.1 seconds to 1.3 seconds, with long-tail jitter significantly converging.
Pitfall 3: Streaming Output in the Generation Phase, RAG-Specific Issues
Streaming output in the generation phase seems simple—the LLM supports a streaming API, just use it.
But there's a unique problem in RAG scenarios: citation annotation.
In standard RAG, after the answer is generated, the system appends citations at the end ("[Source: Product Manual A - Page 3]").
With streaming output, the LLM generates and outputs tokens on the fly. During generation, it doesn't yet know where citation annotations should be placed or which sources should be cited.
Solution 1: Append source annotations after streaming output completes.
The LLM streams the answer body; after generation finishes, the system uniformly appends citations.
Simple to implement, but the user has to wait for the complete answer before seeing the sources—not very elegant.
Solution 2: Instruct the LLM in the prompt to annotate inline during generation.
In your answer, whenever you reference information from the provided materials, immediately annotate that sentence with [Source N],
e.g.: Product A's critical illness insurance has a waiting period of 90 days[Source 1], with a deductible of 5000 yuan[Source 2].
The LLM annotates inline during generation, and citations appear alongside the answer text in the streaming output.
Slightly more complex to implement (need to maintain the mapping between source numbers and chunks in the prompt), but offers a better user experience.
Production problem with Solution 2: LLM's source annotations are unreliable
There's a pitfall not mentioned in the article but must be handled in production: LLMs sometimes hallucinate source numbers. You pass in 3 chunks, and it might write [Source 4], [Source 5], or format them as [Source 1], [Source1], (Source 1)—all over the place.
Post-processing for streaming output needs to specifically handle this:
- Validity check: After streaming output ends, use regex to extract all
[Source N], filtering out hallucinated citations where N exceeds the number of chunks; - Format normalization: Use regex to unify various format variants into a standard format before rendering to the user;
- Fallback plan: If source annotation formats are detected to be chaotic (e.g., multiple formats in the same paragraph), degrade to Solution 1, appending a unified source list at the end.
This logic isn't complex to write, but if you don't write it, it will definitely fail upon launch—the issues QA finds are basically these formatting problems.
Pitfall 4: Streaming Progress Feedback, User Experience During Retrieval
The retrieval phase can't truly be streamed (generation must wait for retrieval to complete), but progress information can be pushed to the user in real-time.
Implementation of progress feedback: SSE (Server-Sent Events)
The server pushes progress events to the client in real-time via an SSE long connection:
event: progress
data: {"stage": "query_rewrite", "message": "Understanding your question...", "progress": 10}
event: progress
data: {"stage": "retrieval", "message": "Retrieving relevant documents...", "progress": 40}
event: progress
data: {"stage": "rerank", "message": "Selecting the most relevant content...", "progress": 70}
event: token
data: {"token": "Acc", "cumulative": "Acc"}
event: token
data: {"token": "ording", "cumulative": "According"}
What the user sees:
[Understanding your question... 10%]
[Retrieving relevant documents... 40%]
[Selecting the most relevant content... 70%]
According to Product Manual A for critical illness insurance, the waiting period is... (streaming output)
Psychological research shows that with progress feedback, the acceptable waiting time for users is 2~3 times longer than without feedback. Even if retrieval still takes 3 seconds, seeing the progress bar move completely changes the waiting experience.
SSE has several easy-to-hit pitfalls in production:
Pitfall 1: Nginx's default buffering breaks SSE.
Nginx buffers upstream responses by default, only forwarding to the client when the buffer is full. SSE requires real-time, event-by-event pushing; buffering turns it into batch sending, completely nullifying the progress bar effect.
The solution is to disable buffering for the relevant route in the Nginx config:
location /api/sse {
proxy_pass http://backend;
proxy_buffering off;
proxy_cache off;
proxy_set_header X-Accel-Buffering no;
}
Miss this config, and it works fine in local testing but breaks immediately upon deployment—because local dev doesn't go through Nginx.
Pitfall 2: Event deduplication on client reconnection.
The SSE protocol has a built-in reconnection mechanism. When the client disconnects, it automatically reconnects and carries the Last-Event-ID header, telling the server which event to resume from.
If the server doesn't handle this header, it will push everything from the beginning again after reconnection. The user sees the progress bar reset to zero and start over, or receives duplicate tokens.
The solution is to add an id field to each event. The server records the maximum event_id sent, and upon reconnection, resumes sending from events after that id:
id: 42
event: token
data: {"token": "ding"}
Pitfall 3: Connection stickiness under load balancing.
SSE is a long connection. All events for the same session must go to the same backend instance (because the event sequence state is in memory).
If the load balancer uses round-robin per request, a reconnection after disconnect might route to a different instance, which won't find the original session state.
The solution is to configure sticky sessions based on session ID, or store the event sequence state in Redis so any instance can resume sending.
Pitfall 5: Error Handling in Streaming Output
If an error occurs mid-stream after streaming output has started (network interruption, LLM timeout, empty retrieval results), handling is more complex than for regular requests.
Error Scenario 1: Error during the retrieval phase.
Streaming output hasn't started yet. Handle it like a regular error: return an error message to the user.
Error Scenario 2: Error mid-generation.
Streaming output has already started; the user has seen part of the answer. Then the LLM errors out or times out.
You can't just return an error, because the user has already seen partial content. An abrupt interruption feels terrible.
Handling: Append "[Answer generation interrupted, please retry]" after the already-output content, and provide a "Continue Generating" button on the frontend to trigger a retry.
Note that "resuming from the interruption point" is practically infeasible in production—LLM generation is non-deterministic. A retry will almost certainly not follow the original path, and forcibly concatenating will produce semantic breaks. The actual practice is to regenerate from scratch, but pass the already-output partial content as context to the LLM to maintain as much consistency as possible, rather than true breakpoint continuation.
Error Scenario 3: Problematic content detected in the streaming output.
Content safety checks are usually done on the complete answer. With streaming output, the answer is still being generated. How to handle this?
Plan A: Buffer the complete answer first, run the safety check, and only stream it to the user after it passes (sacrificing the TTFT advantage of streaming).
Plan B: Use lightweight real-time content filtering (keyword filtering), detecting issues during generation. Immediately interrupt and append a warning if triggered.
These two plans are essentially a trade-off between safety and experience, with no universal answer. Risk-sensitive businesses (finance, healthcare) usually choose Plan A, preferring to sacrifice some TTFT rather than let problematic content reach the user before the check completes. Scenarios with lower content risk can choose Plan B, using lightweight filtering as a fallback, with full safety checks running asynchronously and intervening in subsequent sessions if issues are found.
Prioritizing End-to-End Latency Optimizations
Combining all the optimization points discussed, ranked by benefit/cost ratio:
First Priority: Streaming output in the generation phase
Low implementation cost, huge TTFT benefit. Almost every system should do this.
Second Priority: Retrieval parallelization (Vector + BM25 in parallel)
Medium implementation cost, reduces retrieval phase latency by 40%+. Significant benefit. Remember to add soft timeout protection and concurrency rate limiting simultaneously, otherwise peak traffic might make things worse.
Third Priority: Streaming progress feedback
Medium implementation cost. Doesn't change actual latency but significantly improves perceived user experience. Worth doing. Remember to handle the three pitfalls: Nginx buffering, disconnection reconnection, and load balancing. Otherwise, it works in the test environment but fails in production.
Fourth Priority: Conditional query rewriting
Low implementation cost. Saves 500ms~1 second for simple queries, no impact on complex queries.
Fifth Priority: Dynamically skipping Rerank
Low implementation cost. Saves 200~500ms for high-quality retrieval result scenarios, but the threshold must be calibrated on your own dataset; you can't borrow numbers from others.
How to Answer in an Interview
If an interviewer asks "How did you implement streaming output?" or "How do you optimize user waiting experience?", structure your answer like this:
First, distinguish between TTFT and end-to-end latency. Clearly explain the difference between the two, and that the reason TTFT is high in RAG scenarios is the retrieval pipeline preceding LLM generation. This demonstrates systematic understanding of latency issues.
Then, talk about retrieval phase optimizations. Parallelizing vector retrieval and BM25, the waiting strategy for parallel execution (soft timeout + concurrency rate limiting), conditional query rewriting, dynamically skipping Rerank. Mention specific performance numbers.
Next, discuss streaming in the generation phase. The RAG-specific problem—how to handle citation annotations, the pros and cons of the two solutions, and the production pitfalls of Solution 2 (source hallucination, post-processing for unstable formats).
Then, talk about progress feedback. The SSE implementation, focusing on the three production details: Nginx buffering, disconnection reconnection, and load balancing. Mentioning these earns significant bonus points.
Finally, discuss error handling. Why streaming breakpoint continuation doesn't work in production, the two approaches for content safety checks and their applicable scenarios. This shows you've considered real production constraints.
One Last Thing
Regarding streaming RAG, many people think adding a stream=True parameter counts as having implemented streaming output.
In reality, latency optimization in RAG scenarios is a systemic problem—retrieval parallelization, conditional query rewriting, progress feedback, error handling. Every link has room for optimization, and each has its own pitfalls.
But more importantly, when articulating these, you need to clearly explain the trade-offs behind each decision: why the soft timeout threshold is set that way, how the score cutoff for skipping Rerank is calibrated, why Nginx buffering must be disabled for SSE. Articulating the trade-offs lets the interviewer know you've actually done it, not just memorized conclusions from an article.