No Vector DB Needed: Search-First RAG Inside an Agent Loop
A complete RAG setup here is one ~90 MB ONNX model, a regex tokenizer, and matrix multiplication — teams grafting on vector databases are paying for infrastructure the workload doesn't need. The agent's search behavior is shaped by two sentences in the tool description, which is where the effort actually goes.
RAG's only novel piece is judging relevance. An embedding model reads each note chunk and emits a 512-number vector; cosine similarity between the question's vector and every chunk ranks candidates without any shared words. Indexing and querying must run through the same embedding model, so the entire corpus fits in one numpy array — chunking at level-2 headings, 8 chunks for the sample notes — and a vector database only becomes necessary around a million chunks.
Three gates lose answers: chunks cut wrong, the right chunk never retrieved, or the right chunk ranked out. The honest debugging move is to print the retrieved chunks before touching the prompt. Vector search alone blurs error codes like 401 and 429 into near-identical meaning, so a keyword path counts literal term hits and the two rankings merge by RRF, 1/(60+pos) added per side. Rerank models exist but cost one read per chunk, so they're skipped at this scale.
Turning search into an agent tool changes the flow from fixed one-shot lookup to model-driven retrieval. The tool description tells it to rephrase when results are thin and split multi-part questions; on a two-part question about context loss it searched four times, rephrased "forget" into "context window length limit" on its own, cited sources, and admitted when notes had one incomplete answer. The tradeoffs are real: more turns cost more, and retrieved notes are data, not instructions — a note saying "ignore earlier instructions" can get followed.
A plain numpy array beats a vector database until roughly a million chunks; most setups adopt infrastructure before the data justifies it.
The tool description's phrasing controls how thoroughly the agent searches — retrieval "quality" at the agent level is genuinely a prompting problem.
Since similarity scores lack a pass line, retrieval always returns its top three regardless of whether an answer exists; the system must be able to say "not found".
Embedding models treat error codes as near-synonyms — the 0.003 gap between 401 and 429 shows why exact-match paths survive alongside semantic search.
The debugging discipline of inspecting retrieved chunks before rewriting prompts puts retrieval accountability before prompt-tuning instincts.
The article's keyword path for exact codes draws direct agreement. No-answer scenarios warrant evaluation of their own, separate from cases where retrieval returns results. Exact-code and number lookup rests on that same keyword fallback, which the design already provides.
No-answer scenarios also deserve dedicated evaluation.
Yes.
Code retrieval indeed needs a keyword fallback.