RAG End-to-End: Why a Fund Agent Still Gives Unreliable Answers
RAG deployments in finance and other regulated domains face hard requirements on evidence provenance, permission revocation, and versioning. This walkthrough shows where those requirements actually break in real systems, so engineers can stop chasing recall numbers and start instrumenting the retrieval-to-answer chain for the failures that matter.
RAG systems fail in ways that recall metrics never capture. This piece traces the full path from document ingestion to a final researched answer, using a real fund-announcement question: “What exactly changed in 2023, and can later materials explain that change?” Returning three seemingly relevant documents to the model does not guarantee a correct answer, because the newest file may not be the direct evidence for a historical event, and a keyword match does not mean the right clause applies.
The system separates four distinct chains: knowledge construction (ingesting and versioning materials), online retrieval (permission-filtered candidate ranking), evidence consumption and delivery (what the model actually reads and cites), and evaluation/governance (where to debug the first evidence loss). Each chain has its own failure modes. Permission filtering must happen before Top-K selection, not after, to avoid leaking private notes or squeezing out authorized candidates. Retrieval hits, externalized content locators, and real reads of the original text are three different levels of evidence — only the last one supports a verifiable citation.
The piece also covers versioned updates, the difference between “currently not accessible” and “explicitly asked to delete retained copies,” and a decision table for when to use full RAG versus plain lexical search. An open teaching repository exposes a minimal lexical-retrieval implementation, but the full private system’s architecture is not reproduced there. The goal is not to route every question through more steps, but to ensure that for questions that genuinely need retrieval, the chain preserves the fact of what was found, read, and supported — and what remains unknown.
The recurring failure in RAG is not missing chunks but missing evidence: the system often cannot prove that a specific fragment was actually read and bound to a claim at answer time.
A reliable answer requires distinguishing 'file exists' from 'fully searchable', 'ranked high' from 'supports the claim', and 'tool executed' from 'model used it correctly' — these are separate gates that a single manager cannot own.
Permission revocation and historical replay create a time-of-check/time-of-use problem: content may be readable at retrieval, disclosable at submission, and re-displayable later under different rules, so eligibility must be rechecked at the point of actual disclosure.
The value of hybrid retrieval is task-dependent, not inherent; adding vector search, embeddings, and a reranker imposes real latency and maintenance costs that only pay off when a measured gap appears in a specific stage.
Engineering retrieval metrics like Recall@K are useful for debugging algorithm stages, but they sit far from the final user outcome because model consumption and semantic verification sit between them.