跪拜 Guibai
← Back to the summary

RAG for AI Knowledge Bases: The Hands-On Checklist That Makes It Work

RAG remains the mainstream way to build AI knowledge bases: documents are first chunked, then both vector and keyword indexes are created and stored; at query time, relevant fragments are retrieved, and the LLM answers with sources. The model does not "remember" your materials, and this content is never written into model parameters. Retrieval quality largely decides answer quality: if the retrieved fragments are wrong, a fluent answer is still wrong.

Typical Structure

One pipeline is enough:

  1. Data ingestion: documents, wikis, tickets, code, web pages. Only accept content that is still maintained and has an owner.

  2. Cleaning and chunking: strip headers/footers and duplicate paragraphs; split by headings and paragraphs. Start with chunk sizes around 300-800 characters, breaking at heading and paragraph boundaries; adjacent chunks overlap by about 10%-20% to avoid cutting a sentence in half.

  3. Indexing: each chunk gets both a vector index and a keyword index (BM25). Chinese text must be tokenized before keyword indexing. Use the same embedding model for both ingestion and querying; switching models requires re-embedding the entire library. Attach metadata to each chunk: source, department, effective date, permissions, document type.

  4. Retrieval: first filter by the asker's permissions at query time, then perform hybrid retrieval. Retrieve ten to twenty candidates first, then use a reranking model to compress to 3-5 passages.

  5. Generation: give these passages verbatim to the model along with the question; require it to answer only from the source and to cite references. If nothing hits, or the retrieval score is too low, respond directly with "not in the material" and do not let the model fill in.

  6. Update: when the source document changes, re-chunk it and replace the old fragments with new ones. Expired documents must be able to be taken offline. If you only append and never delete, the library will contain two versions of the same content.

Vector retrieval can sit in Milvus, Qdrant, or the pgvector extension of Postgres. It is just one piece of storage in this pipeline; switching databases rarely improves results on its own.

Choose the Setup by Scale

| Company internal knowledge base | Use self-hostable platforms like Dify, FastGPT, RAGFlow, AnythingLLM, plus unified account and permission management. Documents stay in Feishu, Confluence, Git, where the editable original text lives; the knowledge base stores chunks and indexes for retrieval. | | Deep customization | Write ingestion and retrieval with LlamaIndex or LangChain, put vectors in pgvector or Qdrant, and connect generation to your existing model APIs. |

Gemini Notebook stores a copy of your document, with data on Google's side; it is fine for individual experimentation. Company materials should go on the self-hosted platforms above; check the current version's license and whether the original text leaves your domain. If Dify needs multiple workspaces, verify your version's license terms first.

It is recommended to first run through 50-200 core documents using an off-the-shelf platform. When retrieval accuracy, citations, or permission control fail, then split out and build custom. If you start custom from day one, most time goes into parsing PDFs and operations.

What Actually Drives Quality

Recommended Starting Sequence

  1. Choose a scope, such as "development conventions" or "manual for a product", don't dump all company documents in at once.

  2. Use a self-hostable platform like Dify or RAGFlow to connect these documents; enable hybrid retrieval and citations.

  3. Use it yourself for two weeks and record the questions it gets wrong. First check chunking, titles, and whether the source is outdated. Such errors are not fixed by changing models.

  4. After error causes converge, add scheduled sync and an evaluation set. If all trial users can see these documents, permissions can be added later; if documents have classification levels, add permission filtering from the very beginning.

  5. Once this scope stabilizes, replicate it to the next knowledge domain. Index different domains separately; it is easier to manage than one giant library.

Common failures: pouring everything in, no owner updating the content, and treating citations as optional. A knowledge base is a retrieval system; original document quality and update cadence decide usefulness more than model selection.