The Full Industrial RAG Pipeline: Build, Retrieve, Evaluate, and Operate

—— Build · Retrieve · Evaluate · Operate, Full Business Coverage, Straight to Production

Upload documents, chunk, vectorize, retrieve — that's just the "it runs" version of RAG. In real scenarios you'll hit: scanned PDFs that read as blank, chunking that cuts sentences in half, retrieval that returns irrelevant answers. This article breaks down the complete industrial-grade RAG knowledge base workflow into build, retrieve, evaluate three stages, explaining every parameter clearly.

Preface

In "02-Knowledge Base Vector Retrieval", we implemented the minimal RAG loop: extract → chunk → vectorize → cosine retrieval. But that was just the teaching version, far from "usable":

Scanned PDFs extract as blank;
Fixed 300-character chunking splits "结石高发人群为 30~50 岁成年人，儿童则罕见" into two halves, neither retrievable;
A user asks "What's the probability of getting stones at 35?", the document says "结石高发人群为 30~50 岁成年人", the wording doesn't match and the vectors aren't close enough.

These problems correspond respectively to parsing/splitting/indexing during the build phase, recall/reranking during the retrieval phase, and metric measurement during the evaluation phase. Below we expand on these three stages.

Overall Architecture

RAG Complete Flow

Two pipelines, one closed loop, plus a cross-cutting layer of production operations:

Build (offline): Documents → parsing → cleaning/filtering → splitting → vectorization → storage. Run once, reuse repeatedly.
Retrieval (online): Query → query transformation → multi-path recall → Rerank → Top-K → assemble Prompt to generate answer. Runs on every query.
Evaluation (closed loop): Use test sets to measure retrieval and generation quality, badcase feedback, guide parameter tuning and rebuilding.
Production Operations (cross-cutting): Desensitization before storage, semantic caching before retrieval, review/audit on output, operations flywheel after answering — four insertion points corresponding to layer ④ in the diagram, detailed in Part 5.

One core understanding runs through the entire article: Build-phase parameters determine the form in which data enters the database; changing any one requires rebuilding the database. Retrieval-phase parameters can be adjusted at any time per query.

Part 1: Knowledge Base Construction

1. Document Parsing

The goal of parsing is to unify vastly different file formats into plain text. Processing methods differ completely by format:

Format	Processing Method	Typical Tools
TXT / MD	Direct read	`fs.readFileSync` / Python `open`
Word `.docx`	Extract text from decompressed XML	Node `mammoth`, Python `python-docx`
Word `.doc`	Old binary format, must convert first	LibreOffice convert to `.docx`
Excel	Convert sheet by sheet to text (row concatenation or Markdown table)	`pandas`, `openpyxl`
PDF (text-based)	Directly extract text layer	`pdf-parse`, `PyPDF2`
PDF (scanned)	No text layer, must OCR	`PaddleOCR`, `Tesseract`
Images	OCR for text + multimodal large model image understanding	`qwen-vl` etc. generate image descriptions

Our project implementation (utils/rag.js) covers the first three types:

export async function extractText(filePath) {
    if (filePath.endsWith(".pdf")) {
        const parser = new PDFParse({ data: fs.readFileSync(filePath) });
        const { text } = await parser.getText();   // text-based PDF
        return text;
    }
    if (filePath.endsWith(".docx")) {
        const { value } = await mammoth.extractRawText({ buffer: fs.readFileSync(filePath) });
        return value;
    }
    return fs.readFileSync(filePath, "utf-8");      // txt / md
}

Example: A "Stone Diagnosis and Treatment Manual" may simultaneously contain three types of "pages" — page 1 is normal text, page 3 is a scanned photocopy, page 5 is an anatomical diagram of the urinary system. Industrial-grade parsers (like MinerU, LlamaParse, Unstructured) route page by page: text pages directly extract, scanned pages go through OCR, anatomical diagrams go through a multimodal model to generate a descriptive text ("The diagram shows the positional relationship of the kidneys, ureters, and bladder; stones are more commonly found in the kidneys and ureters"), making image content also retrievable as text.

Data cleaning happens after parsing: unify full-width/half-width characters, remove redundant blank lines, deduplicate headers and footers. The consequence of not cleaning is that the same sentence becomes different chunks due to whitespace differences, wasting storage and interfering with similarity.

2. Document Splitting

Splitting determines "how large a semantic unit one vector represents" — it is the build-phase parameter that most affects retrieval quality.

Split Identifier: By What Rule to Cut

Split Method	Description	Applicable Scenarios
By page	One chunk per page	PPTs, contracts where page = semantic unit
By layout	Cut at visual boundaries of headings/paragraphs/tables	Complex-layout PDFs
By heading level	Cut along H1/H2/H3	Markdown, technical docs
By double newline	One paragraph per chunk	Articles, novels
By newline	One line per chunk	Logs, poetry
Custom delimiter	Cut by `---`, `Chapter X`, etc.	Documents with explicit markers
Fixed length	Sliding window cut by character count	Fallback; our handwritten version uses this

Chunk Length and Overlap Ratio

Chunk length (chunkSize): The upper limit of a chunk. Too large → multiple topics squeezed into one vector, semantics "diluted", retrieval imprecise; too small → incomplete sentences, LLM gets fragmented context. Chinese empirical value: 300~1000 characters.
Overlap ratio (overlap): Number of characters overlapping between adjacent chunks (Volcano Engine uses ratio, we use absolute value). Its purpose is to prevent a sentence from being cut exactly at the split point.

Sliding Window Chunking Example

Example analysis: The manual contains the sentence "Epidemiology: stone high-risk group is adults aged 30~50, incidence rate about 10%; rare in children under 10." 55 characters total.

With chunkSize=30 and no overlap, it may be split into "Epidemiology: stone high-risk group is adults aged 30~50, incidence rate about" and "10%; rare in children under 10." — the first half lacks the child comparison, the second half lacks the subject; a query about "probability of a 5-year-old getting stones" matches neither chunk well;
With overlap, there is always one chunk that contains this sentence intact, ensuring recall.

Advanced: Semantic Chunking — not by character count, but by calculating vector similarity between adjacent sentences; where similarity drops sharply is a topic switch point, cut there. Better results but higher cost; a "use when fixed-length tuning hits a wall" technique.

3. Content Processing Rules

Switches during the parsing phase that decide "whether to turn non-text content into text":

Rule	Effect	Cost
Remove whitespace	Clean noise, save chunk length	Almost none
Enable OCR	Text in scanned pages/images becomes retrievable	Slower build, OCR has recognition errors
Image understanding	Multimodal model generates descriptions for diagrams, making chart content retrievable	Calls large model, has cost

Judgment criterion: Will users ask questions about this content? If they'll ask about stone formation sites in anatomical diagrams, enable image understanding; if the manual is entirely text-based, OCR is waste.

4. Content Filtering Rules

Delete content that is "pure noise during retrieval" before storage:

Rule	Why Filter
Filter references	End-of-document reference lists are useless for Q&A but may be falsely hit by "who is the author" type queries
Filter links and email addresses	URLs/emails have no semantics; vectorized they become noise vectors and dilute the semantics of the chunk they're in

Example: A chunk contains "For technical support contact [email protected], see <https://xx.com/help". If not filtered, this chunk's vector gets skewed by the character patterns of the email and URL; queries genuinely seeking the meaning of this passage actually get lower similarity.

5. Chunk Association Information

After chunks are cut from documents, they lose global context — they don't know which file or chapter they belong to. Association information means attaching metadata back to chunk text for joint vectorization:

Example: The original chunk is just "High-risk group is adults aged 30~50, incidence rate about 10%." Out of context, it doesn't stand out when retrieving "What's the probability of getting stones at 35?". After associating the title, it becomes:

[Stone Diagnosis Manual > Chapter 2 Epidemiology > 2.1 Age Distribution] High-risk group is adults aged 30~50, incidence rate about 10%.

"Epidemiology" and "Age Distribution" in the title supplement the chunk with topic semantics, significantly improving hit rate. These metadata are also stored separately for filtering (search only within a specific file) and traceability (cite sources when answering). Our handwritten version stores source in metadata but hasn't yet concatenated it into the vector text — Volcano Engine's two checkboxes do exactly this "concatenate in" action.

6. Index Enhancement Strategies

For the same chunk, build multi-path indexes to improve recall from different angles:

Strategy	Approach	Problem Solved
Sentence-level index	Index each sentence within a chunk separately	Short queries match more accurately at sentence level
Generate hypothetical questions	LLM pre-generates several "questions this chunk can answer" for each chunk, index the questions	User queries are questions; "question vs question" similarity is higher than "question vs statement"
Generate summary	Generate a summary for the chunk and index it	Match at the "gist" level, resistant to phrasing differences
Generate keywords	Extract keywords to build a keyword index	Support hybrid vector + keyword retrieval

Example (hypothetical questions): Chunk content "Stone high-risk group is adults aged 30~50, incidence rate about 10%; rare in children under 10." Pre-generated questions: "What's the probability of getting stones at 35?" "Can children get stones?". When a user asks "What's the probability of getting stones at 35?", the query matches against these pre-generated questions, almost certainly hitting — this is moving the HyDE concept from retrieval phase forward into the build phase.

7. Vector Database Storage

Regardless of which library is used, every stored record has four elements:

Element	Function	Our handwritten version equivalent
`id`	Unique primary key, supports update/delete	`${source}_${i}`
`document`	Original text chunk, returned directly on hit	`text`
`embedding`	Vector, used for similarity calculation	`embedding`
`metadata`	Source, title, etc., supports filtering and traceability	`source` / `index`

{ "entries": [ { "id": "rag-intro.txt_0", "source": "rag-intro.txt", "index": 0,
                 "text": "RAG is Retrieval-Augmented Generation...", "embedding": [0.012, -0.034, "..."] } ] }

During retrieval, iterate all vectors and compute cosine one by one — brute force but transparent. When data volume grows, switch to a real vector database:

Vector DB	Characteristics
Chroma	Easiest to start, Python embedded ready-to-use; Node version requires separate service
LanceDB	Embedded, pure local file, Node-ecosystem friendly
Milvus / Qdrant	Distributed, million-to-billion scale, production workhorses
pgvector	Installed in PostgreSQL, business DB doubles as vector DB
FAISS	Meta's indexing library, often used as underlying engine

The reason they're fast is using approximate nearest neighbor indexes like HNSW / IVF: not comparing one by one, but traversing a "vector navigation graph" to approach the nearest neighbor in a few hops, returning results in milliseconds for millions of records, at the cost of sacrificing a tiny bit of precision (approximate).

Part 2: Knowledge Base Retrieval

Retrieval Pipeline: Query Transformation → Multi-path Recall → Fusion → Rerank → Top-K

1. Query Transformation: HyDE

Problem: User queries are questions ("What's the probability of getting stones at 35?"), the database stores statements; the two have different phrasing forms, vector distance is large.

HyDE (Hypothetical Document Embeddings): First have the LLM write a hypothetical answer out of thin air, then use that answer's vector for retrieval:

Query: What's the probability of getting stones at 35?
LLM hypothetical answer: The high-risk group for stones is adults aged 30~50,
                         35 years old falls right in this range, incidence rate about 10%.
→ Use the "hypothetical answer" vector for retrieval; its phrasing form matches real chunks' statements, similarity greatly increases

Note HyDE does not require the hypothetical answer to be correct — only that its phrasing form and wording are close to real documents. Similar techniques include multi-query expansion (rewrite one question into 3 different phrasings, retrieve separately, merge results), covering the gap between user wording and document wording.

2. Multi-path Recall

Single vector recall has blind spots: vectors excel at semantic similarity, but are insensitive to proper nouns, numbers, and codes. Multi-path recall runs multiple retrieval paths in parallel then fuses:

Path	Excels at	Example
Vector recall	Semantic similarity, synonymous rephrasing	"get stones" ≈ "suffer from urinary calculi"
Keyword recall (BM25)	Exact word match	Precise terms/values like "incidence rate 10%", "30~50 years old"
Metadata filtering	Scope limitation	Only search `source=Stone Diagnosis Manual.pdf`
Enhanced index (Section 6)	Question/summary/sentence-level match	Hit pre-generated hypothetical questions

Example: Query "probability of a 5-year-old getting stones"; the vector path may rank "adult epidemiological data" higher (semantically close), while the BM25 path, because it precisely contains child, ranks the child incidence chunk first. Two paths complement each other, neither misses.

Fusion commonly uses RRF (Reciprocal Rank Fusion): ignore scores (different paths have different score scales), only look at rank; for each document accumulate 1/(k+rank), sort by accumulated value. Simple, no parameter tuning needed, stable results.

3. Retrieval Methods

Similarity Measures (chosen during build phase, written into index configuration):

Measure	Meaning	Applicable
Cosine similarity	Vector angle, 0~1, larger = more similar	Universal default; our handwritten version uses it
Inner product	Vector dot product, also affected by length	Equivalent to cosine when vectors are normalized
Euclidean distance	Spatial straight-line distance, smaller = more similar	Low-dimensional features

Search Methods:

Exact search (brute force): Compute one by one, results absolutely accurate, O(n) slow. Our JSON version is this; imperceptible within a few thousand chunks.
ANN approximate search: HNSW (build navigation graph, approach in a few hops) / IVF (cluster first then search within clusters). Must-use at million+ scale, trading <1% precision for orders-of-magnitude speed.

4. Rerank

Why needed: Vector recall uses bi-encoder — query and chunk are encoded independently then compared for distance; there is no "interaction" between them, coarse ranking precision is limited.

Rerank uses cross-encoder: concatenate query and chunk together into the model, letting every word cross-attend, outputting a fine-grained relevance score. Precision is a tier higher, but much slower — so it can only be placed after recall as a secondary funnel:

Entire DB 100k chunks → Vector recall Top-50 (millisecond-level, coarse) → Rerank fine ranking → Top-5 (hundred-millisecond-level, fine) → Into Prompt

Example: Among 50 recalled chunks, three all talk about "stone incidence rate"; the cross-encoder can read that only one truly answers the "child" incidence rate (containing "rare under 10" data), ranking it first. Common models: bge-reranker-v2-m3 (deployable locally), cohere-rerank (API).

5. Recall Chunk Count and Similarity Threshold

Two retrieval-phase parameters, adjustable anytime without affecting the database:

Top-K (recall chunk count): Number of chunks ultimately fed to the LLM. K too small → incomplete information; K too large → noise enters context, squeezes context window, and may distract the model with irrelevant content. Empirical value 3~5. Note "Recall Top-50 → Rerank → Top-5" has two Ks: the former is recall width, the latter is the number entering the Prompt.
Similarity threshold: Chunks below the threshold are directly discarded. Its purpose is to handle the "knowledge base has no answer" scenario — when all chunks' similarity is below the threshold, let the model answer "I don't know" rather than fabricating a hallucinated answer. The threshold must be calibrated with a test set; setting it arbitrarily risks false kills.

6. Weighted Scoring Control: Medical Probability Example

The vector database itself only returns "similarity", does not calculate "probability". So for requirements like "judge the probability of getting stones based on the knowledge base", where does weighted control sit? The answer is three layers:

① Retrieval-layer weighting (metadata boost). Attach population metadata to chunks at ingestion: population: adults / population: children. During retrieval, the score is no longer pure similarity but a weighted score:

final_score = similarity × weight(whether population matches query subject)

When querying for "5-year-old child", chunks with population=child get boosted, adult-only chunks get demoted — even if the adult epidemiology chunk has higher similarity, the child chunk can flip to the front via weighting.

② Rerank-layer weighting. The relevance score output by the cross-encoder can be linearly combined with the vector score: score = α·vector_score + β·rerank_score, where α/β are manually set weights to balance "semantic closeness" and "precise relevance".

③ Generation-layer weighting (most commonly used in practice). Write "risk factors and weights" as structured text into the knowledge base, letting the LLM score item by item per the Prompt:

[Stone Risk Scoring Rules] Age 30~50 (+4 points); Male (+2 points);
Daily water intake <1L (+3 points); Family history (+2 points).
Total ≥6 is high risk (probability ~10%), <3 is low risk.

Same knowledge base, two query comparisons:

Query	Retrieval Hit	Weighted Reasoning	Conclusion
Probability of stones at 35?	"High-risk group 30~50, incidence ~10%" + scoring rules	Age item +4 points	High probability
Probability of stones at 5?	"Rare in children under 10, incidence <0.5%" (metadata boost flipped up)	Age item not hit, child data as fallback	Low probability

This answers the article's question clearly: Retrieval is responsible for "finding evidence", weighting is responsible for "using evidence" — probability is not calculated by the vector DB, but is the result of metadata boost + rules/LLM weighted reasoning. To make probability controllable and auditable, write weights into the knowledge base's structured rules (③), rather than handing it to a model black box.

7. Data Access Control: Enterprise Knowledge Base Example

Scenario: Employees query the enterprise knowledge base. Compensation policy documents are visible only to the HR department; technical specifications are visible to all. Without access control, any employee asking "What is the company's compensation structure?" can retrieve compensation documents.

The approach is to attach permission metadata to each chunk at ingestion, and do pre-filtering during retrieval:

Chunk metadata: { source: "Compensation Policy.pdf", visible_roles: ["HR"] }
User identity:  { name: "Zhang San", roles: ["Engineering"] }

Retrieval: filter by permission first, then compute similarity
entries.filter(e => e.visible_roles.some(r => user.roles.includes(r)))

Zhang San asks about compensation; the compensation chunk is excluded at the pre-filtering stage, either not retrieved or only public chunks hit, model answers "No relevant information / No query permission"; an HR colleague asks the same question, filter passes, answers normally. Same question, different answers for different identities — this is the effect of access control.

Two implementation points:

Must pre-filter (push permission conditions down into vector retrieval), cannot post-filter (retrieve first then remove unauthorized). Post-filtering has two problems: unauthorized content occupies Top-K slots, causing "answer exists but not retrieved"; also leaks the existence of sensitive documents ("retrieved but you don't have permission to view"). Chroma's where, Milvus's expr, ES's filter all support pre-filtering; our JSON teaching version does filter first then compute similarity.
Permission metadata must be written at ingestion time: chunks themselves don't know who they belong to; the permission system (departments/roles) must be integrated into the build pipeline, becoming part of metadata just like source.

Part 3: Knowledge Base Test Sets and Evaluation

1. How to Build a Test Set

A test set is a group of triples:

{ "question": "What is the probability of a 35-year-old adult getting stones?",
  "ground_truth": "Adults aged 30~50 are the high-risk group for stones, incidence rate about 10%, significantly higher than children",
  "relevant_chunks": ["Stone Diagnosis Manual.pdf#12"] }

Three sources:

Manual annotation: Domain experts write Q&A pairs; highest quality, highest cost; annotate 50~100 items for cold start;
LLM generation + manual review: Let LLM read each chunk and auto-generate questions, humans filter out bad ones; 10x faster;
Online badcase feedback: Real user queries that were answered wrong, added to the test set — this is the fuel for continuously improving the knowledge base.

2. Retrieval-layer Metrics: Is Recall Accurate?

Metric	Meaning	Calculation Example
Hit Rate@K	Proportion of queries where a relevant chunk appears in top K results	8 out of 10 questions have correct chunk in Top-3 → 80%
Recall@K	Proportion of relevant chunks retrieved (one question may have multiple relevant chunks)	4 relevant chunks total, Top-5 retrieves 3 → 75%
MRR	Average of reciprocal rank of the first correct result	First-hit ranks [1,2,1,3] → (1+0.5+1+0.33)/4 ≈ 0.71
NDCG	Builds on MRR by also rewarding "more relevant results ranked higher"	Closer to 1 the better the ranking quality

Example: Hit Rate@3 improving from 60% to 85% means 2.5 fewer errors per 10 queries — this is the quantified benefit of actions like "enable Rerank" or "adjust chunkSize".

3. Generation-layer Metrics: Is the Answer Good?

Even if retrieval is correct, the LLM may still answer wrong. Generation layer looks at three points:

Metric	What it asks
Answer relevance	Is the answer on-topic?
Faithfulness	Does the answer all come from context, any hallucination?
Context relevance	Are the chunks fed to the model all useful?

RAGAS framework's approach is clever: use a large model as judge, decompose "faithfulness" into mechanically verifiable checks — break the answer into individual statements, check sentence by sentence whether each can find support in the context; the proportion is the faithfulness score. TruLens, LangSmith, Dify's built-in evaluation also provide similar capabilities.

4. How to Judge Whether a Knowledge Base is "Optimal"

There is no once-and-for-all optimum, only a better one found through controlled-variable comparison:

Experiment	Comparison	What to watch
chunkSize 300 vs 500 vs 1000	Rebuild three DBs	Hit Rate@K, MRR
topK 3 vs 5 vs 8	Only change retrieval param	Answer relevance vs faithfulness (larger K more noise, faithfulness may drop)
Rerank on vs off	Only change retrieval pipeline	Whether MRR gain is worth the hundred-millisecond latency
Add BM25 path vs pure vector	Change recall structure	Hit Rate for number-type queries

The process is the closed loop in the overall architecture diagram: run test set → check metrics → adjust one variable → run again → keep if metrics improved. Online badcases continuously feed back into the test set; the evaluation set itself is also growing. "Optimal" is an iterative direction, not an endpoint.

Part 4: Existing Framework Overview

Framework/Platform	Positioning	Characteristics
LlamaIndex	Dev library (Python/TS)	Most comprehensive indexing and retrieval abstractions, finest RAG components
LangChain	Dev library	Largest ecosystem, chain orchestration, paired with LangSmith evaluation
RAGFlow	Open-source RAG engine	Deep document parsing (layout/table/OCR) is its strength
Dify	Open-source LLMOps platform	Visual orchestration + built-in knowledge base and evaluation
FastGPT	Open-source knowledge base platform	Out-of-the-box, multi-model access
Volcano Engine / Bailian	Cloud knowledge base services	The parameter pages this article references come from such platforms; parsing/indexing/evaluation fully managed

Selection advice: Learning phase hand-write (what you're already doing) → Quick project delivery use Dify/FastGPT → Deep custom parsing use RAGFlow → Pure code control use LlamaIndex.

Part 5: Business Processes in Enterprise Production Environments

The first four parts covered the "retrieve-answer" main line. In production environments, four layers of business processes wrap around this main line: knowledge production and operations, query-side processing, compliance and security, online operations flywheel. Strung together it's a closed loop: knowledge production → review and publish → build and store → intent routing → retrieve and answer → compliance review → feedback operations → feed back into knowledge.

1. Knowledge Production and Operations

Review and publish flow: Not whoever uploads takes effect. Documents have three states: draft → review → published; only after review approval do they enter the vector database; in finance/healthcare industries this is a hard compliance requirement.
Versioning and incremental updates: When policies are revised, old chunks are deleted by source, new chunks incrementally ingested (this article's "delete by same source then write" is the minimal implementation); in production, version numbers must also be retained, enabling answers like "per the 2025 version of the policy, how was it stipulated?".
Time-based deactivation: Chunks carry valid_until metadata; expired documents are directly filtered during retrieval, avoiding answers based on obsolete policies.
Conflict detection: When new and old documents contradict, either weight by publish time during retrieval, or use LLM during build phase to compare and identify conflicts, pushing to manual adjudication.
Knowledge gap mining: Cluster online queries with "no hit / low similarity", generate a "knowledge-to-supplement list" for operations — far more effective than manually guessing what documents to add.

2. Query-side Process

Intent routing: First classify the question — knowledge base Q&A, chitchat, or need to call a tool (check order, create ticket)? Route to different pipelines.
Multi-turn rewriting: User's second sentence asks "What about children?", must first use history to complete the query into "probability of a 5-year-old child getting stones" before retrieval.
Multi-knowledge-base orchestration: HR base, tech base, product base kept separate; first route to select base then retrieve, rather than stuffing everything into one big base.
Fallback and transfer to human: Similarity all below threshold → say "I don't know" or transfer to human agent / create ticket, rather than forcing an answer; "refusal rate" is a monitored metric in production.
Citation and traceability: Answers must carry sources (document name + page number + original text highlight), users can click to verify; in medical/legal scenarios, answers without citations are effectively unusable.

3. Compliance and Data Security

Data desensitization is the first gate of data security and must happen before chunking and vectorization. Because vectors themselves are a leakage surface — embeddings are computed from the original text; if the vector database is exfiltrated, it's equivalent to the original text being half-exposed; after retrieval hits, the original text also enters the Prompt, and the LLM may regurgitate PII. The pipeline is:

Original text → PII identification (regex for phone/ID numbers, NER for names/addresses,
  medical scenarios also have case numbers) → masking or pseudonymization → then chunk and store

For pseudonymization, note that the same entity must map to the same pseudonym throughout (Zhang San → "Patient A"), otherwise context won't match during retrieval. On the output side (LLM answer), pass through another masking layer, double insurance. This corresponds to "data minimization" under PIPL/GDPR — sensitive data that doesn't need to enter shouldn't enter.

Other supporting measures:

Output review: LLM answers pass through a sensitive-word/compliance review before reaching the user.
Full-chain audit: Who, when, what asked, which chunks hit, what answered — all logged, traceable afterward; this is the evidence chain when disputes arise.
Permission integration: The pre-filtering from Part 2 Section 7 must integrate with unified identity (SSO/RBAC) in production, not maintain its own role table.

4. Online Operations Flywheel

Monitoring metrics: No-result rate, hit rate, thumbs-up/down, answer latency, token cost; track trends daily.
Badcase closed loop: Thumbs-down and transferred-to-human cases enter annotation queue → added to test set → tune parameters / supplement knowledge → pass regression test before going live — the online version of Part 3's evaluation.
A/B and canary: New chunking strategy, new rerank model first deployed to 10% traffic to compare metrics; only roll out fully after winning.
Semantic cache: Similar questions directly return cached answers, no LLM call — see next section.

5. Semantic Cache: Caching "Similar Questions" Too

Traditional caching's key is the query string; "What is the compensation structure?" and "Company compensation structure?" differ by one character and miss. Semantic caching changes the key to the query's vector: new question first computes similarity against cached questions; if similar enough, directly return the stored answer, the entire RAG pipeline (retrieval + rerank + LLM) is saved:

New question → embed → find top-1 similar question in cache DB
  ├─ sim ≥ 0.95 and isolation dimension matches → directly return cached answer (millisecond-level)
  └─ otherwise → go through full RAG → after answer generation, write {query, embedding, answer, sources} into cache

The storage structure is nearly isomorphic to a vector DB: {id, query, embedding, answer, sources, kb_version, scope, created_at}; existing getEmbeddings + cosineSimilarity can be directly reused. But what's truly valuable are four design points, all pitfalls:

① Threshold must be extremely high. Use this article's medical example to see the risk: "What's the probability of a 35-year-old getting stones?" and "What's the probability of a 5-year-old child getting stones?" can have vector similarity above 0.9, but the answers are completely opposite. Set threshold at 0.85, the child's question will hit the adult's cached answer, silently answering wrong, worse than no cache. So semantic cache threshold is generally ≥0.95, and calibrate with "similar form, different meaning" question pairs as a test set; for extra safety: extract key slots (age/subject/document scope) for hard matching, similarity only as auxiliary.

② Cache must be partitioned by isolation dimension — otherwise permissions leak from cache. Continuing the Part 2 Section 7 example: HR asks "company compensation structure" and gets the real answer cached; Zhang San asks the same sentence; without partitioning, it directly hits HR's cache — data access control is bypassed by the cache. So the cache key, besides the vector, must also carry scope (role/tenant) and kb_version (after knowledge base rebuild, version number +1, all old cache invalidated) — the latter simultaneously solves the invalidation problem: document updates don't need precise cache deletion, just bump the version.

③ Not every answer deserves to enter the cache. Only cache "high-confidence" answers: those where retrieval top-1 similarity exceeds threshold and carry citations are stored; "I don't know" fallback answers are not cached (or short TTL), otherwise after knowledge supplementation the cache still blocks; those involving real-time data (orders, inventory) are not cached.

④ In production, generally do two layers. L1 exact match (normalized string hash, Redis, zero cost) → L2 semantic cache (vector, 0.95+) → L3 full RAG. Customer service scenarios have extremely high repeat phrasing rates; two-layer hit rate can reach 30%~60% — semantic caching essentially trades one embedding call (local model ~10ms) for a second-level retrieval + generation. Ready-made solutions include GPTCache, LangChain SemanticCache, Redis vector search; building it yourself in a teaching project actually exposes the principles most clearly.