RAG for AI Knowledge Bases: The Hands-On Checklist That Makes It Work
RAG remains the mainstream way to build AI knowledge bases: documents are first chunked, then both vector and keyword indexes are created and stored; at query time, relevant fragments are retrieved, and the LLM answers with sources. The model does not "remember" your materials, and this content is never written into model parameters. Retrieval quality largely decides answer quality: if the retrieved fragments are wrong, a fluent answer is still wrong.
Typical Structure
One pipeline is enough:
Data ingestion: documents, wikis, tickets, code, web pages. Only accept content that is still maintained and has an owner.
Cleaning and chunking: strip headers/footers and duplicate paragraphs; split by headings and paragraphs. Start with chunk sizes around 300-800 characters, breaking at heading and paragraph boundaries; adjacent chunks overlap by about 10%-20% to avoid cutting a sentence in half.
Indexing: each chunk gets both a vector index and a keyword index (BM25). Chinese text must be tokenized before keyword indexing. Use the same embedding model for both ingestion and querying; switching models requires re-embedding the entire library. Attach metadata to each chunk: source, department, effective date, permissions, document type.
Retrieval: first filter by the asker's permissions at query time, then perform hybrid retrieval. Retrieve ten to twenty candidates first, then use a reranking model to compress to 3-5 passages.
Generation: give these passages verbatim to the model along with the question; require it to answer only from the source and to cite references. If nothing hits, or the retrieval score is too low, respond directly with "not in the material" and do not let the model fill in.
Update: when the source document changes, re-chunk it and replace the old fragments with new ones. Expired documents must be able to be taken offline. If you only append and never delete, the library will contain two versions of the same content.
Vector retrieval can sit in Milvus, Qdrant, or the pgvector extension of Postgres. It is just one piece of storage in this pipeline; switching databases rarely improves results on its own.
Choose the Setup by Scale
| Company internal knowledge base | Use self-hostable platforms like Dify, FastGPT, RAGFlow, AnythingLLM, plus unified account and permission management. Documents stay in Feishu, Confluence, Git, where the editable original text lives; the knowledge base stores chunks and indexes for retrieval. | | Deep customization | Write ingestion and retrieval with LlamaIndex or LangChain, put vectors in pgvector or Qdrant, and connect generation to your existing model APIs. |
Gemini Notebook stores a copy of your document, with data on Google's side; it is fine for individual experimentation. Company materials should go on the self-hosted platforms above; check the current version's license and whether the original text leaves your domain. If Dify needs multiple workspaces, verify your version's license terms first.
It is recommended to first run through 50-200 core documents using an off-the-shelf platform. When retrieval accuracy, citations, or permission control fail, then split out and build custom. If you start custom from day one, most time goes into parsing PDFs and operations.
What Actually Drives Quality
Slim down the source material. Expired policies, drafts, and chat logs crowd out correct answers. One document, one topic; clear titles; structured tables and steps help more than switching to a stronger model.
Hybrid retrieval + metadata filtering. Pure vector search is unstable with product names, ticket numbers, and error codes; keywords fill this gap. Anything that can be filtered by department, product line, or effective date should be filtered before a full-library search.
Permissions applied before retrieval. Filter by the asker's identity at query time. The retrieval results must not contain passages the user is not allowed to see; filtering at generation stage is too late. When an employee changes roles, the next query reads the new identity, no need to rebuild the library for them.
Answers must include sources. Every conclusion corresponds to a link or passage in the original text. If nothing hits, say "not in the material" and do not let the model hallucinate.
Keep an evaluation set. Pick 30-50 real questions, record the expected answer and the documents or passages that should be hit, and add a few questions that are not in the material and a few that the user is not allowed to see. Use the same set to compare changes in chunking, retrieval, and model; separately measure "did the right passage get hit" and "did the answer stay within the source" otherwise you are just guessing.
Distinguish "find a document" from "summarize". Policies, interfaces, and troubleshooting are suitable for retrieval and excerpt; a synthesis across many documents needs a separate summary or topic index, because a single retrieval cannot cover it.
Recommended Starting Sequence
Choose a scope, such as "development conventions" or "manual for a product", don't dump all company documents in at once.
Use a self-hostable platform like Dify or RAGFlow to connect these documents; enable hybrid retrieval and citations.
Use it yourself for two weeks and record the questions it gets wrong. First check chunking, titles, and whether the source is outdated. Such errors are not fixed by changing models.
After error causes converge, add scheduled sync and an evaluation set. If all trial users can see these documents, permissions can be added later; if documents have classification levels, add permission filtering from the very beginning.
Once this scope stabilizes, replicate it to the next knowledge domain. Index different domains separately; it is easier to manage than one giant library.
Common failures: pouring everything in, no owner updating the content, and treating citations as optional. A knowledge base is a retrieval system; original document quality and update cadence decide usefulness more than model selection.