Streaming RAG Isn't a Flag: The Five Latency Traps That Kill User Experience
TTFT is the latency metric users actually feel, and a RAG system that ignores it will be perceived as slow even if end-to-end numbers look good. The concrete techniques here—parallel retrieval with soft timeouts, conditional query rewriting, and SSE progress feedback—are low-cost, high-impact changes that turn a 3-second blank screen into a responsive experience.
A RAG pipeline that returns a full answer in three seconds can still feel broken if the user stares at a blank screen for 2.5 seconds before the first token appears. The core metric is TTFT (Time To First Token), and in RAG systems it gets inflated by a serial chain of query rewriting, vector search, BM25, RRF fusion, and reranking that all must complete before the LLM ever starts generating. The fix is not a single parameter but a systematic reworking of the retrieval pipeline.
Query rewriting can be made conditional—skip it for the 70–80% of queries that are simple single-hop questions with no pronouns, saving 500ms to 1 second. Vector retrieval and BM25 can run in parallel with a soft timeout (e.g., 800ms), cutting retrieval latency by roughly 44% in measured insurance-knowledge-base workloads. Rerank can be skipped entirely when the top retrieval score already exceeds a calibrated threshold, saving another 200–500ms. Meanwhile, SSE-based progress events ("Understanding your question… 10%") keep the user grounded during the unavoidable wait, which psychological research shows doubles or triples acceptable waiting time.
Production deployment surfaces its own set of traps: Nginx buffers SSE streams by default unless `proxy_buffering off` is set, load balancers break sticky sessions needed for long-lived event connections, and LLMs hallucinate citation numbers like `[Source 4]` when only three chunks were supplied. Error handling mid-stream is also messier than it looks—true breakpoint continuation is infeasible because LLM generation is non-deterministic, so the practical fallback is to regenerate from scratch with the partial output passed as context.
Most teams treat streaming as a generation-phase checkbox (`stream=True`) and miss that the retrieval chain is the dominant contributor to perceived latency.
The threshold-tuning problem recurs across every optimization—rewrite-vs-original comparison, skip-rerank scores, soft timeouts—and none of these values are portable across embedding models or knowledge bases; each team must calibrate on its own data.
Parallelizing retrieval without adding concurrency limits and soft timeouts can backfire under peak load, making latency worse than the original serial pipeline.
Nginx's default response buffering is a silent SSE killer that passes local testing but fails in production, making it one of the most common deployment surprises.
True streaming breakpoint continuation for LLM generation is a fantasy in current architectures; the industry's practical answer is regeneration with context, not resumption.