跪拜 Guibai
← All articles
AI Programming

RAG for AI Knowledge Bases: The Hands-On Checklist That Makes It Work

By J船长 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Teams building internal Q&A assistants will avoid months of trial and error by following this concrete recipe: retrieval quality, permission filtering, and an evaluation set matter more than the choice of model or vector database. The cost of getting these wrong is silent misinformation from a system that looks credible.

Summary

RAG knowledge bases follow a single pipeline: ingest maintained documents, clean and chunk them, build both vector and keyword indexes, then at query time filter by the asker's permissions, run hybrid retrieval, rerank candidates down to 3-5 passages, and feed only those passages to the LLM with a strict citation requirement. The model never stores the material; retrieval quality decides answer quality.

The concrete starting point is small: use a self-hostable platform like Dify or RAGFlow on 50-200 core documents before building anything custom. When things go wrong, the usual culprits are stale content, poor chunk boundaries, and missing permission filters, not the choice of vector database or model. An evaluation set of 30-50 real questions, plus out-of-scope and unauthorized ones, must measure both retrieval hit rate and answer fidelity. Over time, separate indexes per knowledge domain beat one giant library.

Takeaways
— Chunk size starts at 300-800 characters, split on heading and paragraph boundaries, with 10-20% overlap between adjacent chunks.
— Each chunk gets both a vector index and a BM25 keyword index; Chinese text must be tokenized first.
— The same embedding model must be used for indexing and querying; changing models requires re-embedding the whole library.
— Retrieval first filters by the asker's current permissions, then does hybrid search, returns 10-20 candidates, and reranks to 3-5 passages.
— Generation feeds the original passages to the model, requires answering only from the source, and forces citations; if nothing hits, reply 'not in the material' instead of hallucinating.
— Source updates trigger re-chunking and replacement of old fragments, never appending duplicates.
— An evaluation set of 30-50 real questions, plus out-of-scope and unauthorized questions, measures whether the right passage was retrieved and whether the answer stuck to the source.
— Start with an out-of-the-box platform (Dify, RAGFlow, AnythingLLM) on 50-200 core documents before building custom infrastructure.
— Document hygiene (removing stale drafts, chat logs) and clear structure matter more than swapping to a stronger model.
— Index different knowledge domains separately rather than pouring everything into one giant library.
Conclusions

The biggest failure mode is dumping everything into the knowledge base without an owner; original document quality and update cadence decide usefulness more than model selection.

Permission filtering must happen before retrieval, because filtering after generation cannot remove sensitive snippets from the model's context.

A single retrieval cannot cover a synthesis across many documents; those cases need a separate summary or topic index, not a bigger search.

Using a ready platform first reveals where the pain points actually are: chunk boundaries, stale content, and mixed precision fail more often than vector database choice.

Concepts & terms
RAG (Retrieval-Augmented Generation)
A pattern where documents are chunked and indexed; at query time, relevant passages are retrieved and passed to the LLM, which answers with citations rather than from parametric memory.
BM25
A keyword-based ranking function that scores documents by term frequency and inverse document frequency; complements vector search for exact identifiers like product codes and error messages.
Hybrid retrieval
Combining vector similarity and keyword ranking (BM25) to improve recall for both semantic and exact matches; usually requires merging and reranking.
Reranking
A second stage that takes the top 10-20 retrieved passages and scores them with a more powerful model to pick the final 3-5 most relevant ones.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗