No Vector DB Needed: Search-First RAG Inside an Agent Loop
Hand-Built Agent (5): Learning to Flip Through Books — RAG, Search First, Then Answer
Part 5 of the "Hand-Built Agent" series, 10 parts in total. Series intro and table of contents | Previous: Grow a memory — what to do when context is full, how to remember the last time
Figure 5-1 Series map. This post is the last one in the blue box: the eyes.
Preface
The previous post ended with a problem: with 200 notes, the little assistant can't read everything every time. When I ask "why does it forget what was said earlier", the notes say "window overflow", which doesn't literally match.
There's really only one thing to do: find the few passages from a big pile of material that are relevant to the question, and put them into the context. Find well and the model answers well; fail to find and it either says it doesn't know or starts making things up.
This has a name: RAG (Retrieval-Augmented Generation): retrieve first, then put the retrieved content into the context and let the model answer accordingly. I used to think RAG was a heavy thing — that you had to learn a vector database first, then a framework. Taken apart, the only genuinely new step is: how to determine "relevance".
This post explains that step clearly, and the assistant moves up to v0.5.
Key points of this post:
- An embedding model turns a passage into a string of numbers; passages with similar meaning have similar numbers. It lets you search material by meaning.
- When the material holds the answer but the model can't answer, check three gates: chunking, retrieval, ranking.
- Make retrieval a tool and let the Agent decide what to search and how many times.
1. Literal Search Fails When You Rephrase
With more material you can't read everything — the previous two posts explained why: it may not fit; even if it fits it's expensive; even if you don't care about cost, the fuller you stuff it the more it overlooks. So you can only pick out a few relevant passages.
How to pick? The easiest idea is keyword matching: compare words from the question against the notes; literal overlap counts as relevant.
That approach fails directly on my example.
| What I asked | What the notes say |
|---|---|
| Why does it forget what was said earlier | The conversation is too long; the earliest messages get pushed out of the window, the model can't see them |
The two sentences say the same thing, yet share no word. "Forget" maps to "can't see", "what was said earlier" maps to "earliest messages". A human sees it at a glance; a program comparing literally can't.
Keywords compare characters; I want to compare meaning. I need a way to compute "similar in meaning".
2. Embedding: Making Meaning Computable
2.1 A passage becomes a point
There's a kind of specialized model called an embedding model. It doesn't answer questions; it does one thing only: reads a passage and outputs a fixed-length string of numbers, say 512 of them. That string is called a vector, and the process is called embedding.
During training it's required to do one thing: two passages with similar meaning produce similar number strings; unrelated ones sit far apart.
Think of the numbers as coordinates on a map. Each passage is a point; points about the same thing cluster together.
Figure 5-2 Points about the same kind of thing cluster together. My question lands next to the "window overflow" chunk even though they share no word.
So "find relevant material" becomes an arithmetic problem: turn the question into a point too, and see which chunks sit closest.
How is "close" computed? The most common is cosine similarity: how aligned two vectors' directions are, yielding a number that's larger when more similar. You don't need to memorize the algorithm; just know it gives a "similarity score".
2.2 It's a different model from the LLM
This is easy to confuse. The large model that answers and the embedding model are two different things. Both were trained on lots of text, but they learned different skills.
| Large model | Embedding model | |
|---|---|---|
| Learns | what to write next | which two passages belong together |
| Input | a full context | a single passage |
| Output | text continuing the writing | a string of numbers |
| Size | very large | usually much smaller; runs on an ordinary computer |
| Role in RAG | reads found material, writes the answer | decides which material is relevant |
Note Many model providers offer embedding APIs similar in calling style to the chat API. The DeepSeek I use doesn't, so below I run an open-source small Chinese model locally, bge-small-zh-v1.5, about 90+ MB, no GPU needed.
3. Two Pipelines: Indexing and Querying
3.1 Overview first
With embeddings, the whole thing splits into two pipelines.
Figure 5-3 The blue line is prepared ahead of time; the red line runs on every question. Both lines must use the same embedding model, otherwise the numbers on either side can't be compared.
- Indexing: cut notes into small chunks, compute a vector for each, store them. No redo if notes haven't changed.
- Querying: compute the question as a vector, compare similarity against every chunk in the index, take the most similar few chunks into the context, then let the model answer.
Why chunk first, instead of one vector per note? Two reasons. A note often covers several things; squeezed into one vector, none of them stands out. Also, what goes into the context are the found passages; a chunk is far shorter than a note, saving space.
3.2 Prepare notes, then chunk
The code below first creates three sample notes, then chunks them. Focus on split_notes: where the cuts fall, what each chunk holds.
Note This post needs three more packages:
pip install numpy onnxruntime tokenizers. The model API environment variables stay the same, see section 6 of the series intro. If you already created the notes folder when running part 2, rename it first so this post's sample notes get generated.
import json, os, pathlib, re, urllib.request
NOTES = pathlib.Path("notes")
SAMPLES = { # three sample notes, one line is one subsection
"context.md": ["## Window overflow\nThe conversation is too long; the earliest messages get pushed out of the window, the model can't see them.",
"## Prefix cache\nThe part of the request that matches the previous one can be reused, faster and cheaper.",
"## Compression\nReplace earlier messages with a summary; details get lost."],
"tools.md": ["## Tool calls\nThe model only outputs which tool it wants to call; my program does the actual work.",
"## Loop\nStop when the model stops requesting tools. max_steps is the safety net preventing endless turns."],
"pitfalls.md": ["## Error 400\nUsually the tool message's tool_call_id doesn't match the request.",
"## Error 401\nKey is wrong, or the environment variable didn't take effect.",
"## Error 429\nToo many requests; wait a few seconds and retry."]}
if not NOTES.exists(): # if no notes folder, write the sample notes first
NOTES.mkdir()
for name, parts in SAMPLES.items():
(NOTES / name).write_text("\n\n".join(parts), encoding="utf-8")
def split_notes():
"""Cut: split at each level-2 heading, each subsection is a chunk, each chunk starts with which note it came from."""
chunks = []
for path in sorted(NOTES.glob("*.md")):
for part in re.split(r"\n(?=## )", path.read_text(encoding="utf-8")):
chunks.append(f"《{path.name}》{part.strip()}")
return chunks
CHUNKS = split_notes()
- The cut lands before each level-2 heading; one subsection is one chunk. A subsection usually covers exactly one thing.
- Each chunk starts with 《filename》. After a chunk is pulled out alone, you can still see where it came from.
- The three notes cut into 8 chunks in total.
3.3 Index and query
Next is embedding, indexing, and querying. Look at the last two lines: indexing is one line, querying is two.
import numpy as np, onnxruntime
from tokenizers import Tokenizer
if not pathlib.Path("bge").exists(): # on first run, download the embedding model, about 90+ MB
os.mkdir("bge")
base = "https://www.modelscope.cn/models/Xenova/bge-small-zh-v1.5/resolve/master/"
for name in ("tokenizer.json", "onnx/model.onnx"):
urllib.request.urlretrieve(base + name, "bge/" + name.split("/")[-1])
tokenizer = Tokenizer.from_file("bge/tokenizer.json")
tokenizer.enable_padding()
tokenizer.enable_truncation(max_length=512)
session = onnxruntime.InferenceSession("bge/model.onnx")
def embed(texts):
"""Embed: turn several texts into several vectors, each vector is 512 numbers."""
enc = tokenizer.encode_batch(texts)
ids = np.array([e.ids for e in enc], dtype="int64")
mask = np.array([e.attention_mask for e in enc], dtype="int64")
out = session.run(None, {"input_ids": ids, "attention_mask": mask, "token_type_ids": np.zeros_like(ids)})[0]
vecs = out[:, 0] # this model uses the output at the first position to represent the whole passage
return vecs / np.linalg.norm(vecs, axis=1, keepdims=True) # normalize length to 1 for easy comparison
VECS = embed(CHUNKS) # index: one vector per chunk, all stored in one array
def vector_search(query):
"""Query: turn the question into a vector too, compare similarity against every chunk, sort high to low."""
sims = VECS @ embed([query])[0]
return [(int(i), float(sims[i])) for i in np.argsort(-sims)]
- The few lines in
embedare this model's fixed usage: cut text into tokens, feed into the model, take the vector out. To switch embedding models, only this function changes. VECSis the "vector library": an 8-row by 512-column array.VECS @ ...computes similarity scores between the question and all 8 chunks at once. All vectors are normalized to length 1, so the multiplication gives exactly cosine similarity.- The model file is downloaded from ModelScope. If a download breaks halfway, delete the bge folder and run again.
Try the question from the preface. vector_search("why does it forget what was said earlier") gives the top three:
0.466 《context.md》## Window overflow The conversation is too long; the earliest messages get pushed out of the window, the model can't see them.
0.418 《pitfalls.md》## Error 400 Usually the tool message's tool_call_id doesn't match the request.
0.415 《pitfalls.md》## Error 401 Key is wrong, or the environment variable didn't take effect.
The top hit is correct, and it shares no word with the question.
But see the other side: the second and third hits have nothing to do with the question, yet score 0.41. Similarity scores have no "passing line"; only rank is usable. Whether or not the library has the answer, it always produces the "three most similar chunks".
Note I didn't use a vector database here. For 8 chunks, one array is enough; tens of thousands are still manageable. At a million chunks, per-chunk comparison starts to strain, and you also need persistence, add/delete, conditional filtering — that's when you need a vector database. Its core job is the same as this array: store vectors and find the nearest few.
4. When Retrieval Misses, Where's the Problem
After using it, the most common situation is: the answer is clearly in the material, yet it can't answer. For the answer to travel from the notes to the model's eyes it must pass three gates; lose any one and it never arrives.
Figure 5-4 Chunk, find, rank — each gate loses answers in its own way.
4.1 Chunked to pieces
There's more than one way to chunk. The easiest is fixed-size cuts, for example every 200 characters. It ignores content; one cut can slice a sentence in half.
Figure 5-5 Chunk 3 on the left, on its own, doesn't show what it's about. On the right every chunk is one complete thing.
Chunk size matters too. Too small — a chunk holds half a sentence, loses surrounding context, can't be understood alone. Too large — a chunk mixes several things, the vector blurs, and it takes space in the context.
Common approaches:
- Cut along the document's own structure; headings and paragraphs are ready-made boundaries.
- Overlap adjacent chunks slightly so a key sentence doesn't fall exactly on the cut.
- Give each chunk its provenance, such as filename and the subsection heading it's under.
4.2 Not found
The chunk with the answer never makes it into the candidates. Common causes: the question and the material phrase things too differently; the question itself is too short or vague; or only the top three chunks are taken and the answer ranks fifth.
Corresponding fixes:
- Fetch more first, for example the top 20–30 chunks, then filter later.
- Complete the question before searching. For something like "how do I solve that", first clarify what "that" is from the conversation so far.
- Add a keyword-retrieval path alongside vectors; section 5 covers it.
4.3 Ranked wrong
The answer enters the candidates but ranks in the back, and gets cut when the top few are taken. Or the front ranks hold several plausible-but-wrong chunks that lead the model astray.
The fix for this gate is called rerank: use vectors to coarsely select 20–30 chunks, then use a more accurate model to score the question paired with each chunk, reorder, and keep only the top few.
Why not use the more accurate model from the start? Because it pairs the question with each chunk and reads both; with a hundred thousand chunks that's a hundred thousand reads — too slow. The embedding model is fast because every chunk's vector is precomputed at index time; at query time only the question needs computing. So the usual flow is two steps: the fast one filters, the accurate one ranks.
Xiaoji v0.5 doesn't rerank; 8 chunks don't need it.
Note When the answer is wrong, don't rush to change the prompt. Print the retrieved chunks and look: if the answer isn't there, it's a retrieval problem — go back to the three gates; if the answer is there and it still answers wrong, only then is it a model/prompt problem.
5. Keywords and Vectors, Used Together
5.1 Each has its weaknesses
Vector retrieval has its weak spots. I ask "what to do when I hit 401", and its top three:
0.581 《pitfalls.md》## Error 401
0.578 《pitfalls.md》## Error 429
0.552 《pitfalls.md》## Error 400
The top hit is right, but only 0.003 ahead of the second. To the embedding model these three chunks mean nearly the same: records about errors. The 401/429 difference is only in the number, and numbers have little "meaning". With more notes, this order can flip anytime.
Keyword retrieval is cleaner here: only one chunk contains "401".
| Vector retrieval | Keyword retrieval | |
|---|---|---|
| Compares | meaning | literal text |
| Good at | finding rephrased content | codes, names, identifiers in code — every character exact |
| Weak at | telling apart content differing by one code | finding rephrased content |
The two approaches' weaknesses complement each other, so real systems usually run both and merge results. That's called hybrid retrieval.
5.2 Merging by rank
How to merge? The scores on each side can't be added directly: one is a similarity score, the other is a count of matched words — different things. The usual approach is to look only at rank: where each chunk placed on each side, with more points the higher the rank, then sum both sides and sort again.
def terms(text):
"""The "words" for keyword retrieval: contiguous letters/numbers/underscores/dots count as one; Chinese characters count in pairs."""
words = re.findall(r"[a-z0-9_.]+", text.lower())
hanzi = re.findall(r"[一-鿿]+", text)
return set(words) | {s[i:i + 2] for s in hanzi for i in range(len(s) - 1)}
def keyword_search(query):
"""Count how many of the question's terms appear in each chunk; chunks with zero don't count."""
hits = [(i, len(terms(query) & terms(c))) for i, c in enumerate(CHUNKS)]
return sorted([h for h in hits if h[1] > 0], key=lambda h: -h[1])
def search_notes(query, k=3):
"""Hybrid retrieval: rank on each side once, more points the higher the rank, take the top-k chunks by total score."""
score = {}
for ranking in (vector_search(query), keyword_search(query)):
for pos, (i, _) in enumerate(ranking):
score[i] = score.get(i, 0) + 1 / (60 + pos)
return "\n\n".join(CHUNKS[i] for i in sorted(score, key=score.get, reverse=True)[:k])
termsis a crude tokenizer: Chinese characters count in pairs of two. A production system uses a dedicated tokenizer plus an algorithm like BM25 to downweight ubiquitous words like "what" and "how".1 / (60 + pos)is "more points the higher the rank". 60 is the constant this approach conventionally uses; it keeps the gap between the top few positions from getting too large. This merge method is called RRF (Reciprocal Rank Fusion).- A chunk that ranks high on both sides naturally has the highest total.
Ask "what to do when I hit 401" again: the 401 chunk is first on both sides, firmly on top. Ask "why does it forget what was said earlier": no chunk hits on the keyword side, so the vectors decide.
6. Making Retrieval a Tool
6.1 Fixed flow vs. model decides
Up to here, the retrieval flow is fixed: a question arrives, search once, hand the results to the model to answer. Many Q&A applications work this way.
Its flaw is rigidity. If the question raises two things, one search only covers one; if the first search misses, there's no second chance; if I just say hello, it still searches.
Xiaoji already has a looping loop. Make retrieval a tool and hand it over: whether to search, what terms to use, whether what's found is enough, whether to rephrase and search again — the model decides all of it.
Figure 5-6 Left: search once and answer. Right: the loop from part 2, with retrieval as just one of the tools.
6.2 Wiring it into the loop
The code below is nearly identical to the loop from part 2. Look at the last two sentences of the tool description.
from openai import OpenAI
client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])
MODEL = os.environ["LLM_MODEL"]
TOOLS = [{"type": "function", "function": {
"name": "search_notes",
"description": "Find the three most relevant passages for query in my notes. If you don't find what you want, rephrase and search again; if the question covers several things, search separately.",
"parameters": {"type": "object", "required": ["query"],
"properties": {"query": {"type": "string", "description": "What to look for, stated as a complete sentence"}}},
}}]
messages = [{"role": "system", "content": "You are Xiaoji, an assistant that helps me organize study notes. Before answering, search the notes with search_notes first, answer only based on what you found, and cite which note it came from; if you can't find it, say so directly."}]
def run(question, max_steps=6):
messages.append({"role": "user", "content": question})
for step in range(1, max_steps + 1):
resp = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
msg = resp.choices[0].message
messages.append(msg)
if not msg.tool_calls:
return msg.content
for call in msg.tool_calls:
print(f" [round {step}] {call.function.name} {call.function.arguments}")
result = search_notes(**json.loads(call.function.arguments))
messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
return "Ran out of steps without finishing."
while True: # press Ctrl+C to exit
print("Xiaoji: " + run(input("Me: ")))
- The
search_notesfrom the previous snippet is used directly as the tool. It returns the three most relevant chunks, not whole notes. - The description says "rephrase and search again" and "search separately for several things". Whether the model searches multiple times depends largely on those two sentences.
- The system prompt requires answering only from retrieved content, citing the source, and saying so if it can't find anything. That lets me verify later.
Put the four code snippets in order into one file and you get Xiaoji v0.5. I asked a two-part question:
Me: When we've chatted a long time it forgets what happened at the start — why is that? Any solutions?
[round 1] search_notes {"query": "why the model forgets things said at the start of a long conversation"}
[round 1] search_notes {"query": "context window length limit"}
[round 2] search_notes {"query": "how to solve forgetting earlier content, what approaches exist"}
[round 2] search_notes {"query": "write important information outside, external memory or paste it again"}
Xiaoji: Cause: window overflow (from 《context.md》)
What the notes say: the conversation is too long; the earliest messages get pushed out of the window, the model can't see them.……
The only relevant approach found: compression (also from 《context.md》)
Replace earlier messages with a summary. But the notes also flag the cost: details get lost.
……(a couple more lines after this, omitted)
It searched four times. First round: searched for the cause, and on its own rephrased my plain language into "context window length limit". After seeing the results, round two searched for solutions. The answer cited sources and said the notes held only this one approach.
I asked something not in the notes: "Do my notes say anything about how to write system prompts?" It searched four rephrasings, then told me no — and didn't invent something.
That's just this one run. Switch models or run again, and the terms and count of searches may differ.
6.3 Costs and things to watch
- Slower and more expensive. Each additional search adds another loop turn. For simple questions and tidy material, a fixed single search is enough.
- Retrieved content is material, not commands. If a note reads "ignore the earlier instructions", the model may actually follow it. Part 8 covers this specifically.
- Memory can be searched this way too. In the previous post, memory.md is read in full at startup. When memory grows, you can chunk and index it the same way, and fetch only what's needed.
Common misconceptions
| What I used to think | What's actually true |
|---|---|
| Doing RAG requires a vector database first | The core is search-then-answer. With little material one array suffices; a database is only needed at scale |
| The embedding model is a large model | They're two models. The embedding model only turns text into numbers; it doesn't answer questions |
| With vector retrieval, keywords are unnecessary | Each has weaknesses. For codes, names, and other exact-match needs, keywords are more reliable; usually both paths are used |
| Smaller chunks retrieve more accurately | Too small loses surrounding context; the chunk alone is unreadable |
| If retrieved, the model can answer correctly | It must pass chunk/find/rank gates. When an answer is wrong, first check whether the retrieved chunks contain it |
| Windows keep growing, so retrieval won't be needed later | Fits doesn't mean sees accurately, and every turn is billed by length |
Summary
- RAG is search-then-answer: find material first, then let the model answer from it. The only genuinely new step is judging relevance.
- The embedding model turns a passage into numbers; similar meaning gives similar numbers, so you can search by meaning.
- The flow is two lines: indexing (chunk, compute vectors, store) and querying (compute vector, compare similarity, take the top few).
- When the material holds the answer but the model can't answer, check three gates: chunked badly, not found, ranked wrong.
- Vectors compare meaning, keywords compare literal text; use both.
- Making retrieval a tool lets the Agent decide what to search and how many times, at the cost of being slower and more expensive.
This version of Xiaoji:
| Xiaoji v0.5 | |
|---|---|
| What's new vs. last version | Notes chunked and indexed; search_notes tool with hybrid vector+keyword retrieval |
| What it can do | Finds relevant notes even when rephrased; rephrases and searches multiple times on its own; cites sources in answers |
| Remaining problems | Tools are piling up, and it's starting to misuse them |
Next
Count Xiaoji's accumulated tools: list_notes, read_note, remember, and now search_notes.
More tools bring trouble. To find some content, should it use search_notes or read_note? When I say "remember this", does it call remember, or should it write into the notes? Even choosing the right tool, arguments can be wrong.
These mistakes aren't all the model's fault. Tool names, descriptions, parameters, and return values are all written for the model; vague writing leaves it guessing. How should tools be designed so it uses them correctly? Next post will give it nimble hands.
Next: Hand-Built Agent (6): Nimble hands — what a good tool looks like, and give it a computer (link to be added when done)
Top 2 of 3 from juejin.cn, machine-translated. The original thread is authoritative.
No-answer scenarios also deserve dedicated evaluation.
Yes.
Code retrieval indeed needs a keyword fallback.