跪拜 Guibai
← All articles
Agent · Backend · Frontend

Permission-Aware RAG: Building a Knowledge Base That Answers with Evidence

By Sailing ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Most RAG demos ignore tenant and role boundaries, which leaks sensitive data across organizations; this guide shows how to enforce access control during retrieval and keep citations verifiable, so knowledge bases can actually be trusted in production.

Summary

A concrete guide walks through building an enterprise knowledge base with RAG using TypeScript, an OpenAI-compatible API, and Qdrant. The central point is that a usable RAG system is not just about vectorization; it's about treating chunks as production data with full metadata, enforcing permissions at retrieval time, and making answers traceable to evidence. The article lays out a minimal end-to-end chain: document cleaning, chunking with overlap, embedding into Qdrant, permission-filtered retrieval, hybrid search with Reciprocal Rank Fusion, and generation with cited references.

Three production concerns dominate the design. First, every chunk carries a document ID, tenant ID, allowed roles, version, and source URL, so updates can find all related slices and stale versions don't leak into answers. Second, retrieval queries filter by tenant, role, and active version before any ranking or generation happens; no fallback to unfiltered full-database search is allowed. Third, the model is instructed to cite evidence markers like [S1] that map back to real retrieval records, and to abstain when evidence is insufficient rather than inventing plausible answers.

The article also emphasizes separating retrieval from generation in evaluation, using 50-100 real questions with expected documents, required and forbidden facts, and abstention flags. It warns against hardcoded similarity thresholds, argues for version switching to be atomic, and recommends adding security tests for cross-tenant leakage and prompt injection. The conclusion is that RAG's difficulty lies not in vectorization but in governance: traceable slices, permission filtering, version management, and honest abstention.

Takeaways
— Chunk metadata must include source URL, document ID, section path, tenant ID, allowed roles, version, and update timestamp from day one.
— Permission filters (tenant, roles, active version) must be applied inside the retrieval query, not after the fact, and never degrade to an unfiltered full collection search.
— For Chinese documents, start with 300-600 character chunks and 10-20% overlap, but tune based on real questions, document structure, and model context length.
— Switching embedding models or output dimensions requires rebuilding the index; mixing vectors from different models breaks retrieval.
— Hybrid retrieval using vector search plus BM25 should merge results by rank (RRF) rather than raw scores, and each branch must apply the same permission filters.
— Generated answers should reference evidence markers (e.g., [S1]) that are bound to actual retrieval records, and the model must abstain when evidence is insufficient.
— Document version updates should be atomic: write new chunks, validate, switch the active version pointer, then clean up old versions.
— Evaluate retrieval and generation separately using 50-100 real questions, checking for correct document hit, position, stale versions, and unauthorized content.
— Permission tests are security boundary tests, not just retrieval quality tests; they must verify cross-tenant isolation and role-based access denial.
— Don't hardcode a similarity threshold like 0.7; score distributions depend on embedding model, metric, corpus, and question type.
Conclusions

The hardest part of RAG is not vectorization but data governance: permissions, versions, and traceability turn a search trick into a trustworthy enterprise system.

Treating retrieved documents as untrusted content and explicitly instructing the model not to execute instructions inside them is a practical defense against prompt injection.

The article makes a strong case for separating retrieval quality from generation quality in evaluation, since many failures come from the model misusing otherwise correct evidence.

Designing version management around atomic switches and explicit effective dates, rather than relying on updatedAt timestamps, prevents conflicting rules from both v2 and v3 being retrieved simultaneously.

Concepts & terms
RAG (Retrieval-Augmented Generation)
A technique that retrieves relevant document chunks as evidence before generating an answer, so the model's response is grounded in and traceable to external sources.
Reciprocal Rank Fusion (RRF)
A method to merge ranked results from multiple retrieval branches (e.g., vector and keyword) by assigning each item a score based on its rank, avoiding incompatible raw score scales.
Payload Index
An index in vector databases like Qdrant that accelerates filtering on metadata fields such as tenantId, allowedRoles, and version, keeping filtered searches fast as data grows.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗