跪拜 Guibai
← Back to the summary

No Vector DB Needed: Search-First RAG Inside an Agent Loop

Hand-Built Agent (5): Learning to Flip Through Books — RAG, Search First, Then Answer

Part 5 of the "Hand-Built Agent" series, 10 parts in total. Series intro and table of contents | Previous: Grow a memory — what to do when context is full, how to remember the last time

Figure 5-1 Series map

Figure 5-1 Series map. This post is the last one in the blue box: the eyes.

Preface

The previous post ended with a problem: with 200 notes, the little assistant can't read everything every time. When I ask "why does it forget what was said earlier", the notes say "window overflow", which doesn't literally match.

There's really only one thing to do: find the few passages from a big pile of material that are relevant to the question, and put them into the context. Find well and the model answers well; fail to find and it either says it doesn't know or starts making things up.

This has a name: RAG (Retrieval-Augmented Generation): retrieve first, then put the retrieved content into the context and let the model answer accordingly. I used to think RAG was a heavy thing — that you had to learn a vector database first, then a framework. Taken apart, the only genuinely new step is: how to determine "relevance".

This post explains that step clearly, and the assistant moves up to v0.5.

Key points of this post:

1. Literal Search Fails When You Rephrase

With more material you can't read everything — the previous two posts explained why: it may not fit; even if it fits it's expensive; even if you don't care about cost, the fuller you stuff it the more it overlooks. So you can only pick out a few relevant passages.

How to pick? The easiest idea is keyword matching: compare words from the question against the notes; literal overlap counts as relevant.

That approach fails directly on my example.

What I asked What the notes say
Why does it forget what was said earlier The conversation is too long; the earliest messages get pushed out of the window, the model can't see them

The two sentences say the same thing, yet share no word. "Forget" maps to "can't see", "what was said earlier" maps to "earliest messages". A human sees it at a glance; a program comparing literally can't.

Keywords compare characters; I want to compare meaning. I need a way to compute "similar in meaning".

2. Embedding: Making Meaning Computable

2.1 A passage becomes a point

There's a kind of specialized model called an embedding model. It doesn't answer questions; it does one thing only: reads a passage and outputs a fixed-length string of numbers, say 512 of them. That string is called a vector, and the process is called embedding.

During training it's required to do one thing: two passages with similar meaning produce similar number strings; unrelated ones sit far apart.

Think of the numbers as coordinates on a map. Each passage is a point; points about the same thing cluster together.

Figure 5-2 Vector space

Figure 5-2 Points about the same kind of thing cluster together. My question lands next to the "window overflow" chunk even though they share no word.

So "find relevant material" becomes an arithmetic problem: turn the question into a point too, and see which chunks sit closest.

How is "close" computed? The most common is cosine similarity: how aligned two vectors' directions are, yielding a number that's larger when more similar. You don't need to memorize the algorithm; just know it gives a "similarity score".

2.2 It's a different model from the LLM

This is easy to confuse. The large model that answers and the embedding model are two different things. Both were trained on lots of text, but they learned different skills.

Large model Embedding model
Learns what to write next which two passages belong together
Input a full context a single passage
Output text continuing the writing a string of numbers
Size very large usually much smaller; runs on an ordinary computer
Role in RAG reads found material, writes the answer decides which material is relevant

Note Many model providers offer embedding APIs similar in calling style to the chat API. The DeepSeek I use doesn't, so below I run an open-source small Chinese model locally, bge-small-zh-v1.5, about 90+ MB, no GPU needed.

3. Two Pipelines: Indexing and Querying

3.1 Overview first

With embeddings, the whole thing splits into two pipelines.

Figure 5-3 Two pipelines

Figure 5-3 The blue line is prepared ahead of time; the red line runs on every question. Both lines must use the same embedding model, otherwise the numbers on either side can't be compared.

Why chunk first, instead of one vector per note? Two reasons. A note often covers several things; squeezed into one vector, none of them stands out. Also, what goes into the context are the found passages; a chunk is far shorter than a note, saving space.

3.2 Prepare notes, then chunk

The code below first creates three sample notes, then chunks them. Focus on split_notes: where the cuts fall, what each chunk holds.

Note This post needs three more packages: pip install numpy onnxruntime tokenizers. The model API environment variables stay the same, see section 6 of the series intro. If you already created the notes folder when running part 2, rename it first so this post's sample notes get generated.

import json, os, pathlib, re, urllib.request

NOTES = pathlib.Path("notes")
SAMPLES = {   # three sample notes, one line is one subsection
    "context.md": ["## Window overflow\nThe conversation is too long; the earliest messages get pushed out of the window, the model can't see them.",
                 "## Prefix cache\nThe part of the request that matches the previous one can be reused, faster and cheaper.",
                 "## Compression\nReplace earlier messages with a summary; details get lost."],
    "tools.md": ["## Tool calls\nThe model only outputs which tool it wants to call; my program does the actual work.",
               "## Loop\nStop when the model stops requesting tools. max_steps is the safety net preventing endless turns."],
    "pitfalls.md": ["## Error 400\nUsually the tool message's tool_call_id doesn't match the request.",
                  "## Error 401\nKey is wrong, or the environment variable didn't take effect.",
                  "## Error 429\nToo many requests; wait a few seconds and retry."]}
if not NOTES.exists():   # if no notes folder, write the sample notes first
    NOTES.mkdir()
    for name, parts in SAMPLES.items():
        (NOTES / name).write_text("\n\n".join(parts), encoding="utf-8")

def split_notes():
    """Cut: split at each level-2 heading, each subsection is a chunk, each chunk starts with which note it came from."""
    chunks = []
    for path in sorted(NOTES.glob("*.md")):
        for part in re.split(r"\n(?=## )", path.read_text(encoding="utf-8")):
            chunks.append(f"《{path.name}》{part.strip()}")
    return chunks

CHUNKS = split_notes()

3.3 Index and query

Next is embedding, indexing, and querying. Look at the last two lines: indexing is one line, querying is two.

import numpy as np, onnxruntime
from tokenizers import Tokenizer

if not pathlib.Path("bge").exists():   # on first run, download the embedding model, about 90+ MB
    os.mkdir("bge")
    base = "https://www.modelscope.cn/models/Xenova/bge-small-zh-v1.5/resolve/master/"
    for name in ("tokenizer.json", "onnx/model.onnx"):
        urllib.request.urlretrieve(base + name, "bge/" + name.split("/")[-1])
tokenizer = Tokenizer.from_file("bge/tokenizer.json")
tokenizer.enable_padding()
tokenizer.enable_truncation(max_length=512)
session = onnxruntime.InferenceSession("bge/model.onnx")

def embed(texts):
    """Embed: turn several texts into several vectors, each vector is 512 numbers."""
    enc = tokenizer.encode_batch(texts)
    ids = np.array([e.ids for e in enc], dtype="int64")
    mask = np.array([e.attention_mask for e in enc], dtype="int64")
    out = session.run(None, {"input_ids": ids, "attention_mask": mask, "token_type_ids": np.zeros_like(ids)})[0]
    vecs = out[:, 0]   # this model uses the output at the first position to represent the whole passage
    return vecs / np.linalg.norm(vecs, axis=1, keepdims=True)   # normalize length to 1 for easy comparison

VECS = embed(CHUNKS)   # index: one vector per chunk, all stored in one array

def vector_search(query):
    """Query: turn the question into a vector too, compare similarity against every chunk, sort high to low."""
    sims = VECS @ embed([query])[0]
    return [(int(i), float(sims[i])) for i in np.argsort(-sims)]

Try the question from the preface. vector_search("why does it forget what was said earlier") gives the top three:

0.466  《context.md》## Window overflow  The conversation is too long; the earliest messages get pushed out of the window, the model can't see them.
0.418  《pitfalls.md》## Error 400  Usually the tool message's tool_call_id doesn't match the request.
0.415  《pitfalls.md》## Error 401  Key is wrong, or the environment variable didn't take effect.

The top hit is correct, and it shares no word with the question.

But see the other side: the second and third hits have nothing to do with the question, yet score 0.41. Similarity scores have no "passing line"; only rank is usable. Whether or not the library has the answer, it always produces the "three most similar chunks".

Note I didn't use a vector database here. For 8 chunks, one array is enough; tens of thousands are still manageable. At a million chunks, per-chunk comparison starts to strain, and you also need persistence, add/delete, conditional filtering — that's when you need a vector database. Its core job is the same as this array: store vectors and find the nearest few.

4. When Retrieval Misses, Where's the Problem

After using it, the most common situation is: the answer is clearly in the material, yet it can't answer. For the answer to travel from the notes to the model's eyes it must pass three gates; lose any one and it never arrives.

Figure 5-4 Three gates

Figure 5-4 Chunk, find, rank — each gate loses answers in its own way.

4.1 Chunked to pieces

There's more than one way to chunk. The easiest is fixed-size cuts, for example every 200 characters. It ignores content; one cut can slice a sentence in half.

Figure 5-5 Two chunking methods

Figure 5-5 Chunk 3 on the left, on its own, doesn't show what it's about. On the right every chunk is one complete thing.

Chunk size matters too. Too small — a chunk holds half a sentence, loses surrounding context, can't be understood alone. Too large — a chunk mixes several things, the vector blurs, and it takes space in the context.

Common approaches:

4.2 Not found

The chunk with the answer never makes it into the candidates. Common causes: the question and the material phrase things too differently; the question itself is too short or vague; or only the top three chunks are taken and the answer ranks fifth.

Corresponding fixes:

4.3 Ranked wrong

The answer enters the candidates but ranks in the back, and gets cut when the top few are taken. Or the front ranks hold several plausible-but-wrong chunks that lead the model astray.

The fix for this gate is called rerank: use vectors to coarsely select 20–30 chunks, then use a more accurate model to score the question paired with each chunk, reorder, and keep only the top few.

Why not use the more accurate model from the start? Because it pairs the question with each chunk and reads both; with a hundred thousand chunks that's a hundred thousand reads — too slow. The embedding model is fast because every chunk's vector is precomputed at index time; at query time only the question needs computing. So the usual flow is two steps: the fast one filters, the accurate one ranks.

Xiaoji v0.5 doesn't rerank; 8 chunks don't need it.

Note When the answer is wrong, don't rush to change the prompt. Print the retrieved chunks and look: if the answer isn't there, it's a retrieval problem — go back to the three gates; if the answer is there and it still answers wrong, only then is it a model/prompt problem.

5. Keywords and Vectors, Used Together

5.1 Each has its weaknesses

Vector retrieval has its weak spots. I ask "what to do when I hit 401", and its top three:

0.581  《pitfalls.md》## Error 401
0.578  《pitfalls.md》## Error 429
0.552  《pitfalls.md》## Error 400

The top hit is right, but only 0.003 ahead of the second. To the embedding model these three chunks mean nearly the same: records about errors. The 401/429 difference is only in the number, and numbers have little "meaning". With more notes, this order can flip anytime.

Keyword retrieval is cleaner here: only one chunk contains "401".

Vector retrieval Keyword retrieval
Compares meaning literal text
Good at finding rephrased content codes, names, identifiers in code — every character exact
Weak at telling apart content differing by one code finding rephrased content

The two approaches' weaknesses complement each other, so real systems usually run both and merge results. That's called hybrid retrieval.

5.2 Merging by rank

How to merge? The scores on each side can't be added directly: one is a similarity score, the other is a count of matched words — different things. The usual approach is to look only at rank: where each chunk placed on each side, with more points the higher the rank, then sum both sides and sort again.

def terms(text):
    """The "words" for keyword retrieval: contiguous letters/numbers/underscores/dots count as one; Chinese characters count in pairs."""
    words = re.findall(r"[a-z0-9_.]+", text.lower())
    hanzi = re.findall(r"[一-鿿]+", text)
    return set(words) | {s[i:i + 2] for s in hanzi for i in range(len(s) - 1)}

def keyword_search(query):
    """Count how many of the question's terms appear in each chunk; chunks with zero don't count."""
    hits = [(i, len(terms(query) & terms(c))) for i, c in enumerate(CHUNKS)]
    return sorted([h for h in hits if h[1] > 0], key=lambda h: -h[1])

def search_notes(query, k=3):
    """Hybrid retrieval: rank on each side once, more points the higher the rank, take the top-k chunks by total score."""
    score = {}
    for ranking in (vector_search(query), keyword_search(query)):
        for pos, (i, _) in enumerate(ranking):
            score[i] = score.get(i, 0) + 1 / (60 + pos)
    return "\n\n".join(CHUNKS[i] for i in sorted(score, key=score.get, reverse=True)[:k])

Ask "what to do when I hit 401" again: the 401 chunk is first on both sides, firmly on top. Ask "why does it forget what was said earlier": no chunk hits on the keyword side, so the vectors decide.

6. Making Retrieval a Tool

6.1 Fixed flow vs. model decides

Up to here, the retrieval flow is fixed: a question arrives, search once, hand the results to the model to answer. Many Q&A applications work this way.

Its flaw is rigidity. If the question raises two things, one search only covers one; if the first search misses, there's no second chance; if I just say hello, it still searches.

Xiaoji already has a looping loop. Make retrieval a tool and hand it over: whether to search, what terms to use, whether what's found is enough, whether to rephrase and search again — the model decides all of it.

Figure 5-6 Fixed flow vs. as a tool

Figure 5-6 Left: search once and answer. Right: the loop from part 2, with retrieval as just one of the tools.

6.2 Wiring it into the loop

The code below is nearly identical to the loop from part 2. Look at the last two sentences of the tool description.

from openai import OpenAI

client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])
MODEL = os.environ["LLM_MODEL"]
TOOLS = [{"type": "function", "function": {
    "name": "search_notes",
    "description": "Find the three most relevant passages for query in my notes. If you don't find what you want, rephrase and search again; if the question covers several things, search separately.",
    "parameters": {"type": "object", "required": ["query"],
                   "properties": {"query": {"type": "string", "description": "What to look for, stated as a complete sentence"}}},
}}]
messages = [{"role": "system", "content": "You are Xiaoji, an assistant that helps me organize study notes. Before answering, search the notes with search_notes first, answer only based on what you found, and cite which note it came from; if you can't find it, say so directly."}]

def run(question, max_steps=6):
    messages.append({"role": "user", "content": question})
    for step in range(1, max_steps + 1):
        resp = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
        msg = resp.choices[0].message
        messages.append(msg)
        if not msg.tool_calls:
            return msg.content
        for call in msg.tool_calls:
            print(f"  [round {step}] {call.function.name} {call.function.arguments}")
            result = search_notes(**json.loads(call.function.arguments))
            messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
    return "Ran out of steps without finishing."

while True:   # press Ctrl+C to exit
    print("Xiaoji: " + run(input("Me: ")))

Put the four code snippets in order into one file and you get Xiaoji v0.5. I asked a two-part question:

Me: When we've chatted a long time it forgets what happened at the start — why is that? Any solutions?
  [round 1] search_notes {"query": "why the model forgets things said at the start of a long conversation"}
  [round 1] search_notes {"query": "context window length limit"}
  [round 2] search_notes {"query": "how to solve forgetting earlier content, what approaches exist"}
  [round 2] search_notes {"query": "write important information outside, external memory or paste it again"}
Xiaoji: Cause: window overflow (from 《context.md》)
What the notes say: the conversation is too long; the earliest messages get pushed out of the window, the model can't see them.……
The only relevant approach found: compression (also from 《context.md》)
Replace earlier messages with a summary. But the notes also flag the cost: details get lost.
……(a couple more lines after this, omitted)

It searched four times. First round: searched for the cause, and on its own rephrased my plain language into "context window length limit". After seeing the results, round two searched for solutions. The answer cited sources and said the notes held only this one approach.

I asked something not in the notes: "Do my notes say anything about how to write system prompts?" It searched four rephrasings, then told me no — and didn't invent something.

That's just this one run. Switch models or run again, and the terms and count of searches may differ.

6.3 Costs and things to watch

Common misconceptions

What I used to think What's actually true
Doing RAG requires a vector database first The core is search-then-answer. With little material one array suffices; a database is only needed at scale
The embedding model is a large model They're two models. The embedding model only turns text into numbers; it doesn't answer questions
With vector retrieval, keywords are unnecessary Each has weaknesses. For codes, names, and other exact-match needs, keywords are more reliable; usually both paths are used
Smaller chunks retrieve more accurately Too small loses surrounding context; the chunk alone is unreadable
If retrieved, the model can answer correctly It must pass chunk/find/rank gates. When an answer is wrong, first check whether the retrieved chunks contain it
Windows keep growing, so retrieval won't be needed later Fits doesn't mean sees accurately, and every turn is billed by length

Summary

This version of Xiaoji:

Xiaoji v0.5
What's new vs. last version Notes chunked and indexed; search_notes tool with hybrid vector+keyword retrieval
What it can do Finds relevant notes even when rephrased; rephrases and searches multiple times on its own; cites sources in answers
Remaining problems Tools are piling up, and it's starting to misuse them

Next

Count Xiaoji's accumulated tools: list_notes, read_note, remember, and now search_notes.

More tools bring trouble. To find some content, should it use search_notes or read_note? When I say "remember this", does it call remember, or should it write into the notes? Even choosing the right tool, arguments can be wrong.

These mistakes aren't all the model's fault. Tool names, descriptions, parameters, and return values are all written for the model; vague writing leaves it guessing. How should tools be designed so it uses them correctly? Next post will give it nimble hands.

Next: Hand-Built Agent (6): Nimble hands — what a good tool looks like, and give it a computer (link to be added when done)

Comments

Top 2 of 3 from juejin.cn, machine-translated. The original thread is authoritative.

亚雷

No-answer scenarios also deserve dedicated evaluation.

十有八七

Yes.

卷土的土

Code retrieval indeed needs a keyword fallback.