Artificial Intelligence · Full Stack · Agent

The Full Industrial RAG Pipeline: Build, Retrieve, Evaluate, and Operate

By wangfpp · Aug 26, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Most RAG tutorials stop at the toy loop. This walkthrough names the exact parameters, failure modes, and production safeguards — from OCR routing to cache isolation — that turn a demo into a system that won't silently leak permissions or hallucinate probabilities.

Summary

The minimal RAG loop — extract, chunk, vectorize, cosine retrieval — breaks the moment it meets scanned PDFs, sentences cut in half, or queries phrased differently than the stored text. A production system splits into an offline build pipeline (parsing, cleaning, splitting, vectorizing, indexing) and an online retrieval pipeline (query transformation, multi-path recall, rerank, Top-K selection), with evaluation metrics continuously feeding badcases back into rebuild decisions.

Build-phase parameters like chunk size, overlap, and whether to concatenate metadata into the vector text determine how data enters the store; changing any one requires a full rebuild. Retrieval-phase parameters — Top-K, similarity threshold, weighting — are adjustable per query. Multi-path recall combines vector search with BM25 keyword matching and metadata filtering, fused via Reciprocal Rank Fusion, then a cross-encoder reranker does fine-grained scoring on the top candidates.

Production wraps this in four cross-cutting layers: PII desensitization before chunking, semantic caching with isolation dimensions to prevent permission leaks, intent routing and multi-turn rewriting on the query side, and an operations flywheel that monitors hit rates, latency, and token cost while feeding badcases back into the test set. Semantic caching trades a 10ms embedding call for a full second-level retrieval+generation, but the similarity threshold must be ≥0.95 to avoid silently returning wrong answers for near-identical queries with opposite meanings.

Takeaways

— Scanned PDFs and images need OCR or multimodal model descriptions before chunking; otherwise they contribute zero retrievable text.

— Chunk size is the single most impactful build parameter: too large dilutes semantics, too small fragments sentences. Chinese text empirically works at 300–1000 characters with overlap to prevent mid-sentence cuts.

— Semantic chunking — splitting where adjacent sentence vector similarity drops sharply — outperforms fixed-length but costs more; use it only after fixed-length tuning stalls.

— Concatenating document title and chapter metadata into the chunk text before vectorization significantly boosts retrieval hit rate by restoring lost global context.

— Pre-generating hypothetical questions for each chunk and indexing those questions moves the HyDE concept into the build phase, making question-to-question matching nearly certain to hit.

— Multi-path recall combines vector search (semantic similarity) with BM25 (exact term matching) and metadata filtering, fused via Reciprocal Rank Fusion which ignores score scales and only uses rank.

— A cross-encoder reranker reads query and chunk together with full cross-attention, producing far finer relevance scores than a bi-encoder; it runs as a secondary funnel on the top 50–100 recall candidates.

— Two Top-K values exist in a rerank pipeline: recall width (e.g., 50 candidates) and the final number fed into the prompt (e.g., 3–5 chunks).

— A similarity threshold discards all chunks below it, enabling a clean "I don't know" response instead of hallucination when the knowledge base lacks the answer.

— Probability scoring is not a vector DB feature; it requires metadata boost at retrieval time, weighted scoring rules written into the knowledge base, and LLM reasoning at generation time.

— Access control must be pre-filtering (pushed down into the vector query), not post-filtering. Post-filtering lets unauthorized chunks occupy Top-K slots and leaks document existence.

— PII desensitization must run before chunking and vectorization because embeddings themselves are a leakage surface; pseudonymization must map the same entity to the same pseudonym throughout.

— Semantic caching requires a similarity threshold ≥0.95 and isolation by role/tenant/kb_version; otherwise near-identical queries with opposite meanings silently return wrong answers, and permissions leak through the cache.

— A production test set combines manual annotation, LLM-generated questions with human review, and online badcase feedback; the evaluation set grows continuously as badcases are fed back.

— Hit Rate@K, MRR, and NDCG measure retrieval quality; faithfulness (checked by decomposing answers into statements and verifying each against context) measures generation quality.

Conclusions

The article's central distinction — build parameters require a full database rebuild when changed, retrieval parameters are adjustable per query — is a practical cost boundary that most RAG guides never state explicitly.

Moving HyDE-style hypothetical question generation from retrieval time into the build phase (indexing pre-generated questions) is a design tradeoff that trades storage and upfront LLM cost for near-certain recall at query time.

The three-layer answer to "where does probability scoring live" — retrieval metadata boost, rerank weighting, and generation-time structured scoring rules — exposes that vector DBs are evidence finders, not reasoning engines; the reasoning must be explicit and auditable.

The semantic cache isolation design (scope + kb_version as part of the cache key) solves both the permission-leakage problem and the cache-invalidation problem in one mechanism, which is cleaner than maintaining separate invalidation logic.

The article treats evaluation not as a one-time gate but as a continuously growing closed loop where online badcases become test cases, making the test set itself a living artifact that improves with usage.

Concepts & terms

HyDE (Hypothetical Document Embeddings)

A query transformation technique where an LLM first generates a hypothetical answer to the user's question, then that answer's vector embedding is used for retrieval instead of the question's vector. It works because the hypothetical answer's phrasing matches the statement form of stored chunks, even if the answer is factually wrong.

RRF (Reciprocal Rank Fusion)

A fusion method for combining results from multiple retrieval paths (vector, BM25, etc.) that ignores raw scores and only uses rank position. Each document accumulates 1/(k+rank) across all paths, then results are sorted by the accumulated value. k is typically 60.

Bi-encoder vs Cross-encoder

A bi-encoder encodes query and document independently into vectors, then compares distance — fast but coarse. A cross-encoder concatenates query and document together and runs full cross-attention — slow but precise. Production pipelines use bi-encoders for broad recall and cross-encoders for fine reranking on top candidates.

Semantic Chunking

A splitting strategy that computes vector similarity between adjacent sentences and cuts where similarity drops sharply, indicating a topic boundary. More accurate than fixed-length chunking but computationally more expensive.

Semantic Cache

A cache keyed by query vector embedding rather than exact string match. A new query's embedding is compared against cached query embeddings; if similarity exceeds a high threshold (typically ≥0.95) and isolation dimensions match, the cached answer is returned directly, bypassing the entire RAG pipeline.

RAGAS

A framework for automated evaluation of RAG systems that uses an LLM as judge. For faithfulness, it decomposes the generated answer into individual statements and checks each one against the retrieved context; the proportion of supported statements is the faithfulness score.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗