Artificial Intelligence

Streaming RAG Isn't a Flag — It's a Full-Stack Latency Problem

By 神奇小汤圆 · Aug 3, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Perceived latency — the gap between sending a query and seeing the first character — determines whether users trust a RAG system or abandon it. The techniques here (parallel retrieval with soft timeouts, conditional rewrites, SSE progress) turn a 3–6 second blank-screen wait into sub-second first-token delivery without replacing infrastructure, but they demand careful threshold calibration and production hardening that most `stream=True`-only setups ignore.

Summary

A RAG pipeline that adds `stream=True` to the LLM call still makes users wait 3–6 seconds for the first token because the entire retrieval chain — query rewriting, vector search, BM25, Rerank — runs synchronously before generation begins. The fix is a systematic overhaul of the pre-generation pipeline. Query rewriting can be skipped for 70–80% of simple queries or run in parallel with raw-query retrieval, with a similarity-score threshold deciding which results to use. Vector and BM25 retrieval can run concurrently with a soft timeout (e.g., 800ms) to cap worst-case latency, though this doubles downstream load and requires semaphore-based concurrency limits. Rerank can be skipped entirely when the top retrieval score already exceeds a calibrated threshold, saving another 200–500ms.

On the user-facing side, Server-Sent Events push progress through each stage (rewriting, retrieving, reranking) so the wait feels shorter even when it isn't. Three production traps trip up most SSE implementations: Nginx response buffering silently breaks real-time delivery, missing `Last-Event-ID` handling causes duplicate tokens on reconnect, and round-robin load balancing routes reconnects to instances that have no session state. For inline citations during streaming, LLMs routinely hallucinate source numbers or produce inconsistent formatting, requiring a post-processing regex pipeline that validates, normalizes, and falls back to appended citations when the output is too messy.

Takeaways

— TTFT (Time To First Token) and end-to-end latency measure different things; RAG's TTFT is high because retrieval runs entirely before generation starts.

— 70–80% of user queries are simple enough to skip LLM-based query rewriting, saving 500ms–1s per request.

— Running rewritten-query retrieval and raw-query retrieval in parallel, then picking the better result by similarity-score threshold, decouples rewriting latency from TTFT.

— Vector retrieval and BM25 retrieval have no mutual dependencies and can run in parallel, cutting retrieval-phase latency by roughly 44% in one measured insurance knowledge-base project.

— Parallel retrieval requires a soft timeout (e.g., 800ms) so a slow path doesn't drag down the whole request, plus semaphore-based concurrency limits to avoid doubling downstream load at peak.

— Rerank can be skipped when the top retrieval similarity score exceeds a model-specific threshold, saving 200–500ms, but the threshold must be calibrated on the project's own embedding model and dataset.

— SSE-based progress feedback makes users tolerate 2–3× longer waits, but Nginx's default response buffering silently breaks real-time delivery unless `proxy_buffering off` is set.

— SSE reconnection handling requires event IDs and server-side tracking of the last-sent ID; without it, users see progress bars reset or duplicate tokens.

— Load-balanced SSE demands sticky sessions or Redis-backed event state; otherwise, a reconnect routed to a different instance loses the session.

— Inline LLM citations during streaming are unreliable — models hallucinate source numbers and produce inconsistent formatting — so a post-processing regex pipeline for validation, normalization, and fallback is mandatory.

— Mid-stream LLM failures can't be resumed from the interruption point because generation is non-deterministic; the practical fix is full regeneration with prior partial output as context.

— Content safety checks during streaming force a trade-off: buffer-then-check (safer, loses TTFT benefit) vs. lightweight real-time keyword filtering with async full checks (riskier, preserves TTFT).

Conclusions

The gap between 'we do streaming' and actually delivering sub-second first tokens is almost entirely in the retrieval pipeline, not the LLM generation call — yet most RAG tutorials stop at the generation flag.

Parallel retrieval with a soft timeout is a cheap latency win, but it quietly doubles peak load on vector stores and Elasticsearch; teams that skip concurrency limits often find parallel execution slower than serial under real traffic.

Every threshold in this optimization stack — the similarity delta for choosing rewritten vs. raw results, the score for skipping Rerank — is model-specific and dataset-specific. Borrowing numbers from a blog post guarantees misconfiguration.

SSE's production failures are almost never in the application code; they're in Nginx config, missing `Last-Event-ID` handling, and load-balancer session routing — infrastructure details that local testing completely hides.

LLM inline citation during streaming is a genuinely hard problem because it asks a non-deterministic system to produce machine-parseable output in a single pass; the post-processing regex pipeline is an admission that the model can't be trusted here, not a temporary workaround.

Concepts & terms

TTFT (Time To First Token)

The latency from when a user sends a request to when the first character of the response appears. In RAG systems, this includes the entire retrieval pipeline before LLM generation begins, making it the dominant component of perceived latency.

End-to-End Latency (E2E Latency)

The total time from request to complete response. It determines system throughput and cost but is less directly felt by users than TTFT.

Query Rewriting

An initial RAG step that reformulates a user's question (e.g., resolving pronouns, expanding terms) for better retrieval, typically by calling an LLM. It adds 500ms–1s of synchronous latency before retrieval starts.

RRF (Reciprocal Rank Fusion)

A method for combining ranked result lists from multiple retrieval sources (e.g., vector search and BM25) into a single fused ranking, typically run after both retrieval paths complete.

SSE (Server-Sent Events)

A standard allowing a server to push real-time event streams to a client over a single HTTP long connection. Used in streaming RAG to send progress updates (retrieving, reranking) and tokens as they are generated.

Rerank

A fine-grained relevance scoring pass applied to the top candidates from initial retrieval, using a more precise (and slower) model to reorder results before they are sent to the LLM.

BM25

A classic lexical retrieval algorithm based on term frequency and inverse document frequency. In RAG, it is often run alongside vector similarity search to improve recall, especially for keyword-heavy queries.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗