Four Steps That Took a RAG Pipeline from 67% to 92% Hit Rate
Build an evaluation system first, don't blindly tune
Most teams set up RAG, ask a few random questions, feel it's "okay," and launch. When problems arise online, they tweak the prompt, test a few more, feel it's a bit better, and call it a day.
This isn't optimization; it's blind tuning. If you don't even know what level your current system is at, how can you judge whether a change is an improvement or a regression?
Three core metrics
| Metric | Meaning | Target Value |
|---|---|---|
| Hit Rate@5 | Proportion of top-5 results that hit the correct chunk | > 85% |
| MRR | Mean of the reciprocal rank of the first correct result | > 0.75 |
| Context Relevance | LLM-assessed usefulness of context (optional) | > 0.8 |
Hit Rate@K: Whether the top-K retrieval results contain the chunk with the correct answer. Hit Rate@5 = 0.67 means 67% of queries had the correct chunk in the top-5.
MRR (Mean Reciprocal Rank): The average of the reciprocal rank of the first correct result. Rank 1 contributes 1.0, rank 3 contributes 0.33. It's stricter than Hit Rate, requiring not just recall but also that the correct result is ranked high.
Context Relevance: How useful the retrieved results are for answering the question, usually requiring LLM-assisted scoring. Annotation cost is high; if the first two metrics are sufficient, you can skip this one for now.
Building an evaluation set: 50 annotated queries
- Construct queries from real user logs or business scenarios to ensure the distribution matches production.
- Manually annotate the correct chunk ID for each query.
- Be sure to include hard-to-retrieve, vaguely worded queries; otherwise, evaluation results will be artificially inflated.
50 queries are enough to see a trend. Expand to 100-200 later for detailed A/B comparisons.
Evaluation code
# evaluate.py - RAG Retrieval Quality Evaluation: Hit Rate@K and MRR
import json
from typing import List, Set
from dataclasses import dataclass
@dataclass
class EvalQuery:
query_id: str
query: str
relevant_chunk_ids: Set[str] # Manually annotated correct chunk IDs
def calculate_hit_rate(eval_queries, retrieval_results, k=5):
"""Calculate Hit Rate@K: whether the correct chunk is in the top-K"""
qid_to_relevant = {q.query_id: q.relevant_chunk_ids for q in eval_queries}
hits = 0
for result in retrieval_results:
if result.query_id not in qid_to_relevant:
continue
top_k = set(result.retrieved_chunk_ids[:k])
if top_k & qid_to_relevant[result.query_id]:
hits += 1
return hits / len(eval_queries)
def calculate_mrr(eval_queries, retrieval_results):
"""Calculate MRR: average of the reciprocal rank of the first correct result"""
qid_to_relevant = {q.query_id: q.relevant_chunk_ids for q in eval_queries}
rr_sum = 0
for result in retrieval_results:
if result.query_id not in qid_to_relevant:
continue
for rank, cid in enumerate(result.retrieved_chunk_ids, 1):
if cid in qid_to_relevant[result.query_id]:
rr_sum += 1.0 / rank
break
return rr_sum / len(eval_queries)
def run_evaluation(eval_queries, retrieval_fn, k=5):
"""Full evaluation: pass in a retrieval function, return a metrics dict"""
results = []
for q in eval_queries:
chunk_ids = retrieval_fn(q.query)
results.append(type('R', (), {'query_id': q.query_id, 'retrieved_chunk_ids': chunk_ids}))
return {
f"hit_rate@{k}": round(calculate_hit_rate(eval_queries, results, k), 4),
"mrr": round(calculate_mrr(eval_queries, results), 4),
}
With an evaluation system in place, every optimization step gets quantitative feedback.
Baseline: What score can pure vector retrieval achieve?
On the 50-query evaluation set, the baseline performance:
| Metric | Pure Vector Retrieval |
|---|---|
| Hit Rate@5 | 67% |
| Hit Rate@10 | 78% |
| MRR | 0.52 |
Pure vector retrieval has several typical problems:
Semantic similarity does not equal relevance. A user asks "business travel reimbursement standards," and vector retrieval might recall "background on establishing business travel expense management policies," which is semantically close but lacks specific figures.
Keyword omission. A user searches for "ISO-27001 certification." If the document writes "Information Security Management System Certification," the embedding might not match. Conversely, even if the document contains the code "ISO-27001," vector retrieval might still miss it.
Inaccurate ranking. The similarity scores of the top-5 results differ by less than 0.02, but the first is the correct answer while the fifth is completely irrelevant.
Investigating the 17 missed cases: 7 were semantically similar but not answers, 5 were keyword omissions, 3 were chunk segmentation issues (attributed to data preprocessing), and 2 were embedding encoding problems. 12 are improvable through retrieval—these are the areas to focus on next.
Step 1: Hybrid Retrieval (BM25 + Vector)
BM25 is a classic term-based retrieval algorithm whose strengths perfectly compensate for vector retrieval's weaknesses: precise keyword matching, proper noun handling, and short query matching. Conversely, BM25 doesn't understand synonyms or semantic similarity. The two methods are highly complementary.
How to merge the two result lists? Use RRF (Reciprocal Rank Fusion):
RRF Fusion Formula: score(d) = Σ 1 / (k + rank_i(d))
d = candidate document
rank_i(d) = document d's rank in the i-th retrieval system
k = smoothing constant, typically 60
It only looks at rank, not score, requiring no normalization, making it much simpler than weighted fusion. Weighted fusion needs to handle scores of two different scales (BM25 can reach tens, cosine similarity is -1 to 1), and the optimal value of α depends on the data and normalization method, making tuning unstable. RRF bypasses these problems directly.
# hybrid_retriever.py - Hybrid Retrieval: BM25 + Vector + RRF Fusion
from rank_bm25 import BM25Okapi
import jieba # Chinese word segmentation
class BM25Retriever:
"""BM25 Retriever (rank_bm25 version, for production consider Elasticsearch)"""
def __init__(self, chunks):
self.chunks = chunks
self.chunk_ids = [c["chunk_id"] for c in chunks]
# Build index after Chinese word segmentation
self.bm25 = BM25Okapi([list(jieba.cut(c["text"])) for c in chunks])
def search(self, query, top_k=10):
tokenized = list(jieba.cut(query))
scores = self.bm25.get_scores(tokenized)
top_idx = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:top_k]
return [{"chunk_id": self.chunk_ids[i], "score": scores[i], "text": self.chunks[i]["text"]}
for i in top_idx]
class HybridRetriever:
"""
Hybrid Retriever: BM25 + Vector, RRF Fusion
RRF Formula: score(d) = Σ 1/(rank_i(d) + 60)
Only looks at rank, not score, no normalization needed, hassle-free
"""
def __init__(self, bm25_retriever, vector_retriever, rrf_k=60):
self.bm25 = bm25_retriever
self.vector = vector_retriever
self.rrf_k = rrf_k
def search(self, query, top_k=10):
candidate_k = max(top_k * 3, 20)
bm25_results = self.bm25.search(query, top_k=candidate_k)
vector_results = self.vector.search(query, top_k=candidate_k)
# RRF Fusion: score by rank, merge results from both paths
rrf_scores = {}
chunk_map = {}
for rank, r in enumerate(bm25_results, 1):
cid = r["chunk_id"]
rrf_scores[cid] = rrf_scores.get(cid, 0) + 1.0 / (rank + self.rrf_k)
chunk_map[cid] = r
for rank, r in enumerate(vector_results, 1):
cid = r["chunk_id"]
rrf_scores[cid] = rrf_scores.get(cid, 0) + 1.0 / (rank + self.rrf_k)
if cid not in chunk_map:
chunk_map[cid] = r
sorted_ids = sorted(rrf_scores, key=lambda c: rrf_scores[c], reverse=True)[:top_k]
return [{**chunk_map[cid], "score": rrf_scores[cid]} for cid in sorted_ids]
Why use RRF? Explained above, no need to repeat here.
After applying hybrid retrieval, re-running the evaluation set:
| Metric | Baseline | Hybrid Retrieval | Improvement |
|---|---|---|---|
| Hit Rate@5 | 67% | 78% | +11pp |
| MRR | 0.52 | 0.63 | +0.11 |
An 11 percentage point improvement—this is the most cost-effective step. 7 of the 12 retrieval bad cases were fixed.
Step 2: Reranking
Hybrid retrieval solved recall, but ranking is still an issue. Both BM25 and vector retrieval are Bi-Encoders. A Cross-Encoder can fill this gap:
| Dimension | Bi-Encoder (Vector Retrieval) | Cross-Encoder (Reranker) |
|---|---|---|
| Encoding Method | Query and Doc encoded separately | Query and Doc concatenated and encoded together |
| Token Interaction | None | Full interaction via Attention layers |
| Speed | Fast (can build index offline) | Slow (cannot be offline, only for reranking) |
| Accuracy | Moderate | Higher |
| Applicable Stage | Full library recall | Fine-ranking of candidate set |
So the typical approach is two-stage: Bi-Encoder recalls top-50 from the full library, Cross-Encoder reranks these 50, and takes the top-5.
# reranker.py - Reranking based on BGE-Reranker-v2-M3
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
class BGEReranker:
"""Cross-Encoder Reranker"""
def __init__(self, model_name="BAAI/bge-reranker-v2-m3"):
device = "cuda" if torch.cuda.is_available() else "cpu"
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = AutoModelForSequenceClassification.from_pretrained(model_name).to(device).eval()
self.device = device
def rerank(self, query, candidates, top_k=5):
"""Rerank candidate results"""
if not candidates:
return []
# Concatenate query and each chunk, then feed into the model
pairs = [[query, c["text"]] for c in candidates]
with torch.no_grad():
inputs = self.tokenizer(
pairs, padding=True, truncation=True,
max_length=512, return_tensors="pt",
).to(self.device)
scores = self.model(**inputs).logits.squeeze(-1).cpu().numpy()
for i, c in enumerate(candidates):
c["rerank_score"] = float(scores[i])
return sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)[:top_k]
# Complete retrieval pipeline: Hybrid retrieval recalls 50 -> Reranker takes top-5
class RAGRetriever:
def __init__(self, hybrid_retriever, reranker, first_stage_k=50, final_k=5):
self.hybrid = hybrid_retriever
self.reranker = reranker
self.first_stage_k = first_stage_k
self.final_k = final_k
def search(self, query):
candidates = self.hybrid.search(query, top_k=self.first_stage_k)
return self.reranker.rerank(query, candidates, top_k=self.final_k)
Reranking model selection: If you have a GPU or acceptable CPU latency, use BGE-Reranker-v2-M3 (good for both Chinese and English, open-source and free). If you don't want to deploy, use the Cohere Rerank API. Run the evaluation first before deciding whether to switch.
Performance after adding reranking:
| Metric | Hybrid Retrieval | +Reranking | Improvement |
|---|---|---|---|
| Hit Rate@5 | 78% | 88% | +10pp |
| MRR | 0.63 | 0.79 | +0.16 |
The MRR improvement (+0.16) is more significant than the Hit Rate improvement (+10pp). The core value of reranking isn't recalling more, but moving the correct result from a lower position to the front.
Step 3: Query Transformation
The first two steps modified the retrieval side, but sometimes the problem lies in the query itself. A user asks "how to reimburse travel expenses," while the document says "employee business trip expense settlement process." BM25 can't match it, and vector similarity is also low.
Three strategies:
Query Expansion: Synonym expansion, expanding "annual leave" to "annual leave, yearly vacation, paid leave." Simple but prone to introducing noise.
Multi-Query: Have the LLM rewrite the query from multiple angles, retrieve for each, and fuse the results with RRF.
HyDE: Have the LLM first generate a hypothetical answer, then use the embedding of that hypothetical answer for retrieval. This "translates" the query into the language of the documents, narrowing the distribution gap.
# query_transformer.py - Query Rewriting: Multi-Query + HyDE
import json
class QueryTransformer:
def __init__(self, llm_client, model="gpt-4o-mini"):
self.llm = llm_client
self.model = model
def multi_query(self, query, num_queries=3):
"""Multi-Query: Have the LLM rewrite the query from multiple angles"""
prompt = f"""Rewrite the following query {num_queries} times from different angles, keeping the semantics unchanged.
Original Query: {query}
Return only JSON: {{"queries": ["rewrite1", "rewrite2", ...]}}"""
resp = self.llm.chat.completions.create(
model=self.model,
messages=[{"role": "user", "content": prompt}],
temperature=0.3,
)
return json.loads(resp.choices[0].message.content)["queries"]
def hyde(self, query, doc_length=200):
"""HyDE: Generate a hypothetical document, use its embedding for retrieval"""
prompt = f"""Based on the following question, write a hypothetical document of about {doc_length} words.
It should look like content that might exist in a knowledge base; it doesn't need to be perfectly accurate.
Question: {query}
Return the document content directly."""
resp = self.llm.chat.completions.create(
model=self.model,
messages=[{"role": "user", "content": prompt}],
temperature=0.5,
)
return resp.choices[0].message.content.strip()
class MultiQueryRetriever:
"""
Multi-Query Retriever: Rewrite query -> Multi-path retrieval -> RRF Fusion -> Reranking
"""
def __init__(self, hybrid_retriever, transformer, reranker=None):
self.hybrid = hybrid_retriever
self.transformer = transformer
self.reranker = reranker
def search(self, query, top_k=5, num_rewrites=3, use_hyde=False):
if use_hyde:
# HyDE: Vector retrieval uses hypothetical doc, BM25 uses original query
hyde_doc = self.transformer.hyde(query)
vector_results = self.hybrid.vector.search(hyde_doc, top_k=30)
bm25_results = self.hybrid.bm25.search(query, top_k=30)
# RRF fuse the two result lists
all_result_sets = [bm25_results, vector_results]
else:
# Multi-Query: Original + rewritten queries each go through hybrid retrieval
rewritten = self.transformer.multi_query(query, num_rewrites)
all_queries = [query] + rewritten
all_result_sets = [self.hybrid.search(q, top_k=30) for q in all_queries]
# RRF fuse all retrieval results
rrf_scores = {}
chunk_map = {}
for results in all_result_sets:
for rank, r in enumerate(results, 1):
cid = r["chunk_id"]
rrf_scores[cid] = rrf_scores.get(cid, 0) + 1.0 / (rank + 60)
if cid not in chunk_map:
chunk_map[cid] = r
sorted_ids = sorted(rrf_scores, key=lambda c: rrf_scores[c], reverse=True)[:50]
candidates = [{**chunk_map[cid], "score": rrf_scores[cid]} for cid in sorted_ids]
if self.reranker:
return self.reranker.rerank(query, candidates, top_k=top_k)
return candidates[:top_k]
Comparison data for each strategy:
| Strategy | Hit Rate@5 | MRR | LLM Calls |
|---|---|---|---|
| Hybrid Retrieval+Reranking | 88% | 0.79 | 0 |
| + Query Expansion | 89% | 0.80 | 1 |
| + Multi-Query | 91% | 0.83 | 1 |
| + HyDE | 90% | 0.82 | 1 |
| + Multi-Query + HyDE | 92% | 0.85 | 2 |
The risks of HyDE must be clearly explained. A user asks "how is the company's severance pay calculated?" HyDE generates a hypothetical document citing Article 47 of the Labor Contract Law, and retrieval works very well. But if the user asks a vague question like "how is that compensation matter handled?" HyDE might generate a hypothetical document about "traffic accident compensation," going completely off track.
Conclusion: The more specific the user's question, the better HyDE performs. For vague questions, Multi-Query is more stable.
Step 4: Context Processing and Generation Optimization
Retrieval has reached a 92% Hit Rate, but directly stuffing chunks into the LLM still causes problems: irrelevant chunks interfere with generation, and Lost in the Middle (LLMs pay less attention to content in the middle of long contexts).
# context_processor.py - Context Processing: Deduplication, Reordering, Prompt Construction
import numpy as np
class ContextProcessor:
"""Preprocessing of retrieval results before feeding into the LLM"""
def __init__(self, dedup_threshold=0.90, max_chunks=5, max_length=3000):
self.dedup_threshold = dedup_threshold
self.max_chunks = max_chunks
self.max_length = max_length
def deduplicate(self, chunks, embedding_model):
"""Deduplicate based on embedding similarity"""
if len(chunks) <= 1:
return chunks
texts = [c["text"] for c in chunks]
emb = embedding_model.encode(texts)["dense_vecs"]
emb = emb / (np.linalg.norm(emb, axis=1, keepdims=True) + 1e-8)
kept, removed = [], set()
for i in range(len(chunks)):
if i in removed:
continue
kept.append(chunks[i])
for j in range(i + 1, len(chunks)):
if j not in removed and np.dot(emb[i], emb[j]) > self.dedup_threshold:
removed.add(j)
return kept
def reorder_for_attention(self, chunks):
"""Reorder to mitigate Lost in the Middle: put the most relevant at both ends"""
if len(chunks) <= 2:
return chunks
left, right = [], []
for i, c in enumerate(chunks):
(left if i % 2 == 0 else right).append(c)
return left + right[::-1]
def truncate(self, chunks):
"""Control total length"""
total, result = 0, []
for c in chunks:
if total + len(c["text"]) > self.max_length:
remaining = self.max_length - total
if remaining > 100:
result.append({**c, "text": c["text"][:remaining] + "..."})
break
result.append(c)
total += len(c["text"])
return result
def build_rag_prompt(query, chunks):
"""Construct a grounding prompt: restrict answers to context + cite sources"""
context = "\n\n".join(
f"[Document {i+1}]\n{c['text'][:500]}" for i, c in enumerate(chunks)
)
return f"""You are a knowledge base Q&A assistant. Answer the question based on the following documents.
Requirements:
1. Only answer based on the content of the following documents
2. If the documents are insufficient to answer, clearly state "Based on the available information, I cannot answer"
3. Cite information sources in the format [Document X]
4. Keep it accurate and concise
Retrieved Documents:
{context}
User Question: {query}
Answer:"""
Context processing doesn't contribute to retrieval metrics, but answer accuracy improved from about 83% to about 88%. Grounding instructions and citation requirements significantly reduce hallucinations.
Optimization Summary
| Stage | Hit Rate@5 | MRR | Answer Accuracy | Incremental Gain |
|---|---|---|---|---|
| Baseline (Pure Vector) | 67% | 0.52 | ~60% | - |
| + Hybrid Retrieval | 78% | 0.63 | ~70% | +11pp |
| + Reranking | 88% | 0.79 | ~80% | +10pp |
| + Query Transformation | 92% | 0.85 | ~83% | +4pp |
| + Context Processing | 92% | 0.85 | ~88% | +0pp (+5pp generation) |
Diminishing marginal returns are clear. Hybrid retrieval and reranking are must-dos, a 21 percentage point improvement with uncomplicated implementation. Query transformation is optional; it works well but increases latency. Context processing doesn't contribute to retrieval metrics but does to generation quality; it's low cost and recommended to always do.
One-sentence summary: Hybrid retrieval and reranking are must-dos, a 21 percentage point improvement with uncomplicated implementation. Query transformation is optional; it works well but increases latency, requiring a trade-off evaluation.
Optimization Path Selection:
1. Build evaluation set (50 annotated queries) — Don't start without evaluation
2. Add hybrid retrieval (BM25+Vector+RRF) — Expected +10pp, low complexity
3. Add reranking (Cross-Encoder) — Expected +10pp, medium complexity
4. Add query transformation (Multi-Query/HyDE) — Expected +3~5pp, requires LLM calls
5. Context processing (Dedup+Reorder+Prompt) — Answer accuracy +3~5pp, low complexity
6. Continuous iteration: Expand evaluation set, periodic regression testing, A/B test for launch
Pitfalls in Production
Retrieval failure fallback: A 92% Hit Rate means 8% still aren't retrieved. When the top score is below a threshold, directly tell the user "No relevant content found," which is better than generating hallucinations. Determine the threshold based on the score distribution in the evaluation set.
Offline evaluation does not equal online performance: A 50-query evaluation set is just a signal. You must run A/B tests at launch, looking at user behavior metrics (likes/dislikes, follow-up questions, copy actions). Sometimes offline metrics improve while online metrics decline.
Latency trade-offs: Multi-Query + HyDE with two LLM calls takes 1-3 seconds, reranking 200-800ms, generation 1-3 seconds, totaling potentially 4-5 seconds. Optimization directions: use a smaller model for query rewriting, reduce candidate count for reranking, parallelize multi-path retrieval, and use streaming output.
Continuous optimization: Add online bad cases to the evaluation set and run periodic regression tests. Every time you change the retrieval strategy, run the full evaluation set to ensure no regression.
Summary
Reviewing the optimization path: Build evaluation system (quantify the problem) -> Hybrid retrieval (solve recall) -> Reranking (solve ranking) -> Query transformation (solve query-document mismatch) -> Context processing (solve generation quality). Each step solves a different problem, and they complement each other.