LLM · Algorithm · Architecture

RAG Isn't Just a Knowledge Base: The Full Retrieval Pipeline from Embedding to Rerank

By 得物技术 · Jul 23, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Teams treating RAG as just a vector database and an LLM will hit a ceiling fast. The difference between a demo and a production system lies in the retrieval pipeline—chunk overlap, hybrid search, query rewriting, and reranking—and getting any one of these wrong means the LLM generates plausible-sounding answers from irrelevant or broken context.

Summary

Retrieval-Augmented Generation is often reduced to 'chat with your PDFs,' but its real architecture is a chain of independent stages that each make or break answer quality. Chunking strategy determines whether an answer gets severed across boundaries or diluted into noise. Embedding models turn semantic similarity into a vector-distance math problem, but they are blind to exact codes and proper nouns, which is why BM25 keyword search remains essential. HNSW indexing makes approximate nearest-neighbor search fast enough for millions of vectors by layering a navigable small-world graph, trading a controlled amount of recall for orders-of-magnitude speed gains.

The full online pipeline runs Query Rewrite (users rarely ask well-formed questions), Metadata Filtering (multi-tenant guardrails), multi-path Recall (dense vectors + sparse keywords), RRF fusion (rank-based merging that avoids score-scale mismatches), and a Cross-Encoder Reranker that reads query and document together to fix the coarse ranking mistakes a Bi-Encoder can't catch. Each stage addresses a specific failure mode, and skipping any one of them degrades the final answer the LLM generates.

Takeaways

— Chunking is the highest-stakes preprocessing step: too small severs answers across boundaries, too large dilutes semantics into an unsearchable average vector.

— Embedding models are trained with contrastive learning and optimized for cosine similarity; using a different distance metric breaks the scoring assumptions baked into the model.

— Dense vectors capture semantic meaning but fail on exact codes and proper nouns; BM25 or sparse vectors are required for literal keyword matches.

— HNSW combines skip-list layering with greedy graph search to drop vector search complexity from O(N) to O(log N), with ef_search as the only runtime-tunable accuracy knob.

— Query Rewrite is often the single highest-ROI optimization: users type vague, context-heavy queries that no retriever can handle without LLM preprocessing.

— Metadata filtering must run before vector search in multi-tenant systems, or documents from other tenants will leak into results.

— RRF fuses multi-path recall results by rank, not raw score, sidestepping the impossible task of normalizing cosine similarity and BM25 scores onto a common scale.

— A Cross-Encoder Reranker reads the query and candidate document together, catching relevance distinctions that a Bi-Encoder Embedding model loses when it compresses each side into independent vectors.

— The 'Lost in the Middle' phenomenon means stuffing more chunks into an LLM context window can hurt answer quality; placing the most relevant chunks at the edges matters more than total count.

Conclusions

RAG's retrieval pipeline is a stack of independent, replaceable components, yet most framework defaults treat chunking and recall as afterthoughts. The real engineering work is tuning each stage to the specific document structure and query patterns of a domain.

Embedding models and reranker models are often discussed as interchangeable, but they solve opposite halves of the retrieval problem. Embedding models are optimized for speed and recall; reranker models are optimized for precision and read query-document pairs jointly. Conflating the two leads to systems that are either slow, inaccurate, or both.

The article's framing of memory as a first-class retrieval source alongside static documents is under-discussed in Western RAG literature. User-specific dynamic knowledge—preferences, conversation history, short-term state—requires separate attribution, conflict-resolution, and expiry logic that a standard vector DB does not provide.

Concepts & terms

Embedding

The process of mapping text into a fixed-length vector of floats using a trained neural network. Models are trained with contrastive learning to place semantically similar text close together in vector space, turning 'meaning similarity' into a distance calculation.

HNSW (Hierarchical Navigable Small World)

An approximate nearest-neighbor algorithm that builds a multi-layer graph. Upper layers have few nodes and long-range edges for fast coarse navigation; the dense bottom layer enables fine-grained search. Combines skip-list layering with greedy graph traversal to achieve O(log N) search complexity.

RRF (Reciprocal Rank Fusion)

A method for merging ranked results from multiple retrieval paths (e.g., vector search and BM25) without needing to normalize raw scores. Each document's final score is the sum of 1/(k + rank) across all paths, where k is a smoothing constant (typically 60).

Cross-Encoder Reranker

A model architecture that processes a query and a document together as a single input pair, outputting a relevance score. Unlike Bi-Encoders (which encode query and document independently), Cross-Encoders can detect fine-grained mismatches—like 'VPN installation' vs. 'VPN password reset'—but are too slow to run over the full corpus and are used only on top-N recall candidates.

Lost in the Middle

A documented LLM behavior where the model pays significantly less attention to content in the middle of its context window. In RAG, this means dumping many retrieved chunks into the prompt can backfire; placing the most relevant chunks at the beginning or end improves answer quality more than adding more chunks.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗