跪拜 Guibai
← All articles
Artificial Intelligence · Backend · Frontend

No Vector DB Needed: Search-First RAG Inside an Agent Loop

By 十有八七 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

A complete RAG setup here is one ~90 MB ONNX model, a regex tokenizer, and matrix multiplication — teams grafting on vector databases are paying for infrastructure the workload doesn't need. The agent's search behavior is shaped by two sentences in the tool description, which is where the effort actually goes.

Summary

RAG's only novel piece is judging relevance. An embedding model reads each note chunk and emits a 512-number vector; cosine similarity between the question's vector and every chunk ranks candidates without any shared words. Indexing and querying must run through the same embedding model, so the entire corpus fits in one numpy array — chunking at level-2 headings, 8 chunks for the sample notes — and a vector database only becomes necessary around a million chunks.

Three gates lose answers: chunks cut wrong, the right chunk never retrieved, or the right chunk ranked out. The honest debugging move is to print the retrieved chunks before touching the prompt. Vector search alone blurs error codes like 401 and 429 into near-identical meaning, so a keyword path counts literal term hits and the two rankings merge by RRF, 1/(60+pos) added per side. Rerank models exist but cost one read per chunk, so they're skipped at this scale.

Turning search into an agent tool changes the flow from fixed one-shot lookup to model-driven retrieval. The tool description tells it to rephrase when results are thin and split multi-part questions; on a two-part question about context loss it searched four times, rephrased "forget" into "context window length limit" on its own, cited sources, and admitted when notes had one incomplete answer. The tradeoffs are real: more turns cost more, and retrieved notes are data, not instructions — a note saying "ignore earlier instructions" can get followed.

Takeaways
— Relevance judging is the only genuinely new step in RAG; everything else is established plumbing.
— Embedding models and chat LLMs are separate models; the embedding one only produces numbers.
— The sample uses bge-small-zh-v1.5, a 90+ MB Chinese embedding model run locally with onnxruntime; no GPU required.
— Chunking at structural boundaries (level-2 headings) with source filenames prepended yields 8 chunks from 3 notes.
— Cosine similarity scores have no absolute threshold; only the ranking is usable.
— Vector-only retrieval ranks 401/429/400 notes within 0.003 of each other; keyword search handles exact codes.
— Hybrid merge uses RRF with the 1/(60+pos) scoring; 60 keeps top ranks from dominating too hard.
— Rerank pairs the question with each chunk and reads both — too slow to be the first stage; two-step coarse-then-fine is standard.
— A fixed retrieval flow searches once; the agent tool version lets the model choose queries, counts, and rephrases.
— The system prompt pins answers to retrieved content with citations and requires a direct "not found".
— On one two-part question the agent searched four times and rephrased the user's wording; results vary by model and run.
— Costs: extra search means extra loop turns, slower and more expensive; simple well-formed inputs are better served by single-shot retrieval.
Conclusions

A plain numpy array beats a vector database until roughly a million chunks; most setups adopt infrastructure before the data justifies it.

The tool description's phrasing controls how thoroughly the agent searches — retrieval "quality" at the agent level is genuinely a prompting problem.

Since similarity scores lack a pass line, retrieval always returns its top three regardless of whether an answer exists; the system must be able to say "not found".

Embedding models treat error codes as near-synonyms — the 0.003 gap between 401 and 429 shows why exact-match paths survive alongside semantic search.

The debugging discipline of inspecting retrieved chunks before rewriting prompts puts retrieval accountability before prompt-tuning instincts.

Concepts & terms
RAG
Retrieval-Augmented Generation: retrieve relevant passages from a store, insert them into the model context, then let the model answer from them.
Embedding model
A model that maps a passage to a fixed-length vector; passages with similar meaning land close together in vector space.
Vector
A fixed-length list of numbers representing a passage's meaning, used for similarity math.
Cosine similarity
A score measuring how aligned two vectors' directions are; larger means more similar.
Chunking
Cutting documents into small self-contained units, for example per heading, before embedding so vectors stay meaningful and context stays small.
Rerank
A second, more accurate model that scores the question against each candidate chunk pair to reorder a coarse shortlist.
Hybrid retrieval
Running vector and keyword search in parallel and merging results to cover both semantic and exact-match queries.
RRF (Reciprocal Rank Fusion)
Combining multiple rankings by adding 1/(60+position) per list and sorting by total; no score normalization needed.
From the discussion

The article's keyword path for exact codes draws direct agreement. No-answer scenarios warrant evaluation of their own, separate from cases where retrieval returns results. Exact-code and number lookup rests on that same keyword fallback, which the design already provides.

— No-answer scenarios deserve their own evaluation, distinct from normal retrieval cases.
— Keyword fallback is indispensable for code and number retrieval, confirming the article's hybrid design.
Featured comments
亚雷

No-answer scenarios also deserve dedicated evaluation.

十有八七

Yes.

卷土的土

Code retrieval indeed needs a keyword fallback.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗