Four Steps That Took a RAG Pipeline from 67% to 92% Hit Rate
Most teams launch RAG with blind prompt tweaking and no quantitative baseline. This optimization sequence gives a repeatable, low-complexity path—hybrid retrieval and reranking alone deliver 21 points of Hit Rate improvement—and makes clear which steps are must-dos versus optional latency trade-offs.
Starting from a pure vector-search baseline scoring 67% Hit Rate@5 and 0.52 MRR on a 50-query evaluation set, a four-stage optimization pushes retrieval to 92% Hit Rate@5 and 0.85 MRR. The first stage adds BM25 keyword retrieval fused with vector results via Reciprocal Rank Fusion, fixing 7 of 12 bad cases and gaining 11 percentage points. A cross-encoder reranker then re-scores the top-50 candidates, lifting MRR by 0.16—more than Hit Rate—by pulling correct chunks to the front. Query rewriting with Multi-Query and HyDE closes the gap on vocabulary mismatch between user questions and document language, adding another 4 points at the cost of one or two extra LLM calls. Finally, context deduplication, reordering to combat Lost in the Middle, and grounding prompts improve answer accuracy from roughly 83% to 88% without changing retrieval metrics.
The work also surfaces the real production trade-offs: a 92% Hit Rate still means 8% of queries fail, so a score-threshold fallback that says "no relevant content found" prevents hallucination. Offline gains don't guarantee online wins—A/B testing against user behavior remains essential. And the latency budget tightens fast when Multi-Query and HyDE each add an LLM round-trip, pushing total response time toward 4–5 seconds unless smaller models or parallel retrieval are used.
Hybrid retrieval and reranking are the highest-leverage steps: together they deliver 21 percentage points of Hit Rate improvement with no LLM calls and moderate implementation complexity.
MRR is a stricter and more revealing metric than Hit Rate because it penalizes correct chunks that appear late in the result list—reranking's true value shows up in MRR, not just recall.
Query rewriting is where the optimization shifts from deterministic retrieval to probabilistic LLM calls, introducing a latency-vs-accuracy trade-off that each team must evaluate against their own p95 latency budget.
The 8% residual failure rate after all optimizations is a reminder that retrieval is never perfect; a production system needs a fallback policy, not just better ranking.
Offline evaluation sets drift from production distribution over time, so continuous bad-case logging and periodic regression testing are as important as the initial optimization sprint.