1. Why RAG is Needed — The Innate Limitations of LLMs

When asking ChatGPT questions related to an enterprise's internal systems, a common situation occurs: it provides an answer that seems logically coherent but is actually completely wrong. Many people have encountered similar situations.

When coding, if you ask "How do I use the v2.3 API of this framework," it might provide a long-obsolete v1.x syntax; if you ask "What is the refund process for the company's internal coupon system," it will fabricate a process out of thin air.

The core flaws of LLMs can be summarized in three points:

• Training data has a cutoff date: GPT-4's knowledge stops at the moment its training was completed and cannot know about new content that emerged afterward.

• Hallucinations occur when encountering knowledge blind spots: It will not honestly reply "I don't know" but will forcibly fabricate a seemingly reasonable answer.

• Inability to recognize private enterprise knowledge: Internal documents, codebases, business rules, and team conventions—this type of content does not appear in public training data.

RAG (Retrieval-Augmented Generation) is precisely designed to solve these problems. The idea is very simple: let the LLM retrieve information first, then provide an answer based on the retrieved content.

User Query → Retrieve relevant external information → Stuff the information into the prompt → LLM answers based on the information

Note the wording above; I wrote "retrieve relevant external information," not "retrieve from a knowledge base." This is because RAG essentially does expand the LLM's reasoning context. The retrieved content can come from any source: a PDF file, a chat history, a search result, a web link.

RAG is the straw, and the retrieval source is the water in the cup. The cup can be a knowledge base (a collection of documents), a search engine (the internet), or a single file. The straw and the contents of the cup are independent of each other.

Many people equate RAG with "building a knowledge base Q&A system," which is a one-sided, narrow understanding. However, in practical application scenarios, the most common and highest-value combination is indeed RAG + Knowledge Base. Therefore, the following text will focus on this main line.

Where Does Knowledge Come From

If RAG is to retrieve a knowledge base, what exactly is stored in the knowledge base? To rephrase the question—where does the "knowledge" within an organization actually reside? It can actually be divided into two categories: static shared knowledge and dynamic personal memory.

Static Shared Knowledge

Static shared knowledge is relatively stable information shared by a team.

Enterprise Documents: Product documents, technical proposals, meeting minutes, and review reports in Feishu Docs, Confluence, Notion, and Google Docs. This is the most structured and easily retrievable knowledge.

Code Repositories: Source code, READMEs, API documentation, and changelogs in Git repositories. For example, asking "How does this microservice integrate with unified authentication"—the answer might be in a repository's README.

Chat Records: Technical discussions in Slack, Feishu groups, and DingTalk. For example, "Who fixed that NPE last time? How was it fixed?"—the answer might be in a group chat record from three months ago.

Rules and Regulations: Attendance rules, reimbursement processes, and holiday policies in HR systems. This content has a fixed tone, and the ways of asking questions are also clear.

Dynamic Personal Memory

There is another type of content rarely mentioned in the traditional definition of RAG, but it has a huge impact on user experience—the user's personal preferences, habits, and historical behavior. For example: Every time Zhang San asks "Help me write an interface," the phrasing is consistent, but he wants an interface in Go + Gin style, while Li Si needs Java + Spring Boot. This is not a specification written in a document but Zhang San's own technical preference. Another example in customer service: A user previously reported "The page loads too slowly," and now asks "It's still lagging." The system needs to recognize that "still" refers to the previous problem, rather than treating it as a completely new issue.

This is Memory—dynamic knowledge that varies from person to person and changes over time. Unlike documents, which are "the same content visible to everyone," memory is "each user has their own exclusive, independent storage."

Both static shared knowledge and dynamic personal memory rely on retrieval to match relevant content, but memory additionally presents three types of engineering challenges: Subject Attribution (determining which user the memory belongs to), Conflict Resolution (the logic for choosing when old and new memories contradict each other), and Lifecycle Management (preferences may be long-term valid, but short-term states like "recently learning Rust" might expire the next month). This article focuses on the knowledge base as the main line; the engineering details of memory implementation will be discussed in a separate article.

The RAG Pipeline: Offline Storage, Online Search

The RAG pipeline is divided into two stages. The following explanation uses the knowledge base scenario as an example; the memory retrieval architecture is similar, with differences mainly in access strategies, to be discussed in a later article.

What happens in the offline stage: Documents in the knowledge base are cut into paragraphs, each paragraph is calculated into a vector using an Embedding model, and stored in a vector database. This step runs periodically, pre-computed and ready, so there is no waiting during queries.

Document → Chunking → Embedding (Vectorization) → Index Writing (Store in Vector DB)

Online stage: User query → Query vectorization → Vector search for Top-K relevant documents → Feed documents and query together to the LLM → LLM generates an answer.

(Simplified version) User Query → Embedding → Recall (ANN + BM25) → RRF Fusion → Rerank Fine-ranking → LLM Generates Answer

The complete chain also includes Query Rewrite and Metadata Filter, which will be expanded upon later.

Query Rewrite → Metadata Filter → Recall (ANN + BM25) → RRF Fusion → Rerank Fine-ranking → LLM Generates Answer

Whether RAG works well depends not on the LLM, but on "searching accurately." If the retrieved information is wrong, no matter how strong the LLM is, it's garbage in, garbage out.

So the problem to solve is actually: How to search to find content with similar meaning, "not just" literal matches?

2. Embedding: Turning Text into Computable Data

To search for "meaning," we first need to make it computable for computers. Computers only recognize 0s and 1s; they don't know how to compare "similarity of meaning."

What Embedding does is turn text into vectors, making semantic similarity a mathematical calculation problem.

Looking back at the RAG pipeline, the Embedding in the offline stage and the Embedding in the online stage are the same thing—mapping text into a vector space, then finding the nearest neighbors in that space.

From "Searching Literal" to "Searching Meaning"

Traditional database search is literal matching. Using SQL LIKE '%programming language%' to search:

✅ "The user's programming language is Go" → Hit, because the characters "programming language" appear ❌ "The user writes Go and Python" → Missed, the characters "programming language" are absent, but the meaning is completely relevant ❌ "The user's tech stack is Go" → Missed, similarly semantically relevant but literally different

This is like going to a library to look up materials, and the librarian can only match characters in the book title one by one—you search for "programming," and he can't find a book titled "Coding."

To make computers understand "meaning," a completely new way of representing data is needed—Embedding (Vector Embedding).

What is a Vector

A vector is an ordered set of numbers, also a column vector in linear algebra—except the dimensions go from two or three to hundreds or thousands (mainstream Embedding models commonly have 768, 1024, or 1536 dimensions).

[3.0, 4.0] is a two-dimensional vector, [0.12, -0.34, 0.07] is a three-dimensional vector. The value of vectors lies in the ability to compare them mathematically—the "closer" two vectors are, the more similar the things they represent. As for how "closeness" is calculated, that will be expanded upon later when discussing similarity.

In RAG, "vector" refers to a string of floating-point numbers obtained after converting text, for example, 1536 dimensions:

"The user likes Sichuan cuisine" → [0.21, -0.43, 0.07, 0.85, ..., -0.12] ← 1536 numbers

Once text becomes a vector, "Are these two sentences similar in meaning?" becomes "Are these two vectors close in distance?"—turning a linguistic problem into a mathematical one that computers can handle.

So who decides what vector a piece of text should become? This is the job of the Embedding model.

What is Embedding

Embedding is the process of turning a piece of text into a point in a vector space—but the core behind this is a specially trained model at work. It's not simply "each piece of text randomly generates an array."

The whole process is: A piece of text is input into the Embedding model, the model calculates through its internal neural network, and outputs a fixed-dimension vector. Each dimension of this vector is a floating-point number, and together they represent the semantic features of the text.

The key is—this model is specifically trained for semantic alignment. It knows that "Sichuan cuisine," "spicy," and "hot pot" are semantically in the same circle and will place them very close in the vector space. It also knows that "Sichuan cuisine" and "It's raining today" are unrelated and will place them far apart.

In other words, what the Embedding model does is semantic vectorization—turning the fuzzy "similarity of meaning" in human language into precise "closeness of distance" in vector space. Once this transformation is complete, semantic search becomes a purely mathematical operation—just calculate the distance between two vectors.

Let's use two-dimensional vectors to understand the principle. Although real model outputs are 1024 or 1536 dimensions, it's clear at a glance on a two-dimensional plane [Note: The AI-generated schematic in the text has distorted details, but it's sufficient for conveying the idea]:

"The user likes Sichuan cuisine" and "loves spicy food" are very close—the model has learned that "Sichuan cuisine" and "spicy" are semantically related. "It's raining today" has nothing to do with them and is placed in a completely different location.

The same logic applies during search: the query text is also fed into the same model to calculate a vector, then the nearest neighbors are found. For example, searching for "loves strong flavors," the model maps it near (3.2, 3.5)—close to both "Sichuan cuisine" and "spicy," and the results come back.

So the magic of Embedding lies not in the vector itself, but in that trained model—it knows what kind of text should be placed together. How did the model learn this ability?

How Embedding Models Are Trained

The training method is called Contrastive Learning. Imagine you are holding a remote control with two buttons—"Bring Closer" and "Push Away":

In each round, the model fine-tunes its parameters based on the signal. Pulling positive examples closer teaches it "these two sentences are related in meaning," pushing negative examples away teaches it "these two sentences are not related." After billions of rounds, the model internalizes the semantic relationships of language—no one needs to tell it the relationship between "Sichuan cuisine" and "spicy"; it induces it from massive training data on its own.

This is somewhat similar to how we learn a language—no one explains to you that "happy" and "joyful" are synonyms; you naturally learn it by seeing them appear in similar contexts countless times.

The source of training data, in the final analysis, is collecting naturally occurring correlation signals from the internet—search engine click logs (what users searched for and which result they ultimately clicked), adoption records from Q&A communities, parallel corpora (Chinese and English versions of the same content), plus some synthesized by LLMs. In short, it's giving the model endless feedback of "this question is right, this question is wrong."

This also determines the limitation of Embedding: The model is only effective in the domain covered by its training corpus. An English model will degrade when searching Chinese; a general model will be inaccurate when searching medical terminology. Which model to use depends on what domain and language your data is in.

Mainstream Embedding Models:

A common misconception worth clarifying here—Embedding models and large language models like GPT and Claude are not the same thing. Although both are based on Transformers at the bottom layer, their goals and structures are completely different:

Embedding models do compression—condensing the semantics of a piece of text into a vector; LLMs do generation—spitting out text step by step based on context.

In the RAG pipeline, each has its own role: the Embedding model is responsible for offline vectorization and online query vectorization, while the LLM is responsible for finally generating an answer based on the retrieval results.

Dense vs Sparse: Two Types of Vectors, Two Capabilities

Knowing how the model is trained, let's look back at the vectors it produces. The most common output of Embedding models is Dense vectors, but vectors actually come in two distinctly different forms. They search for different things and are complementary, not substitutes for each other.

• Dense (Dense Vector): Every dimension has a value, e.g., [0.12, -0.34, 0.07, ...] × 1536 dimensions. Searching for "database connection pool" can hit "DB connection pool config"—completely different words, but the same meaning. The weakness is insensitivity to proper nouns and codes—searching for a trace ID like "SLB-20250101-abc," semantic vectors are of no help.
• Sparse (Sparse Vector): Only a few dimensions have values, e.g., {wordID_1523: 2.4, wordID_8912: 1.7}. Each dimension corresponds to a word; searching for "SLB" precisely hits all documents containing "SLB." The weakness is that changing the word means it won't be found—"database" and "DB" are completely different dimensions.

By analogy, Dense searches for "meaning," Sparse searches for "literal." A good search system needs both to cooperate. [Note: The Sparse vectors here (like SPLADE) and BM25 discussed below are not the same thing. Sparse vectors are an output form of Embedding models; BM25 is a statistical algorithm that does not produce vectors and uses an inverted index—Lucene and Elasticsearch's default scoring formula is this. Both do similar work (both capture keyword matching). This article uses BM25 as the representative for explanation and does not expand on Sparse vectors.]

Embedding handles the output of vectors, but how long is the "piece of text" fed into the model? How to cut it? This is what the next chapter, Chunking, will discuss.

3. Chunking: How to Cut Knowledge

The pipeline mentioned at the beginning of the article included "Chunking" but didn't expand on it. This step is actually very critical—the poor performance of many RAG projects is fundamentally not due to a bad Embedding model or the wrong vector database choice, but because the chunking strategy was not done well.

Why Cut

Isn't it possible to just feed the entire document directly into the Embedding model? The idea is natural, but reality presents 3 hurdles.

• Embedding models have input length limits. Mainstream models support a maximum of 512 to 8192 tokens. Some models (like e5-mistral-7b) already support over 32k, but a product document can easily be tens of thousands of words; stuffing it full is also a problem.

• Even if the model can handle ultra-long texts, semantic dilution will greatly reduce retrieval effectiveness. A 50-page product document covering dozens of feature points; the vector calculated by the Embedding model becomes an "average" of these dozens of features. This average is not close enough to any single specific feature—a user searching "How long does a refund take to arrive?" will find that the refund only accounts for a fraction of the vector representation, and retrieval simply cannot grasp it.

• Even if retrieved, the content sent to the LLM should not be the entire document. The LLM's context window is limited. Stuffing the entire document in wastes tokens and dilutes the truly relevant content—the LLM needs precise paragraphs, not massive amounts of original text.

Therefore, documents must be cut into appropriately sized paragraphs, each paragraph vectorized individually, and the most relevant segments matched during queries. So, what size is appropriate to cut? This is a dilemma.

• Cut too fine—context breaks, and the answer might be severed. For example, in a refund policy document, a user asks "How long does a refund take to arrive?" and the answer is in the middle of the document. If mechanically cut by fixed length, the answer might exactly straddle the boundary of two chunks. During retrieval, each chunk grabs half, and neither can answer fully.

• Cut too large—back to the semantic dilution problem. A chunk stuffed with 2000 tokens covers three or four topics, and the vector becomes an "average" again, decreasing precision.

An analogy: If a book's table of contents only lists down to the chapter level (too coarse), searching for the specific content of a subsection will definitely fail; if it's detailed down to every natural paragraph (too fine), context is easily lost during search. A good chunking strategy finds the right granularity between these two extremes.

Understanding why to cut, let's look at several common strategies below.

Chunking Strategies

Fixed-Length Chunking + Sliding Window

The most straightforward approach is Fixed Chunking—one chunk every 500 tokens, simple and crude.

Document: [0..500] [501..1000] [1001..1500] ...

Pros: Simple implementation, fast. Cons: Easy to cut in the middle of a sentence, destroying semantic integrity.

An improved solution is the Sliding Window—overlap between chunks:

Chunk 1: [0..500] Chunk 2: [300..800] ← Overlaps 200 with the previous one Chunk 3: [600..1100] ← Continues overlapping

The overlapping area allows sentences crossing boundaries to at least appear completely in one chunk, significantly reducing information loss.

Semantic Chunking

Going a step further, cut according to the document's natural structure—headings, paragraphs, subsections:

`## Refund Rules ← Cut by heading Paragraph 1... ← Chunk 1 Paragraph 2... ← Chunk 2

Return Rules ← New heading, new Chunk

Paragraph 3... ← Chunk 3`

Semantic chunking maintains the semantic integrity of each chunk, avoiding the situation of "sentences being severed." But it requires the document itself to have structure (Markdown headings, HTML tags, etc.); it doesn't work well for plain text.

Parent-Child Chunk

A more common practice in modern RAG systems is Parent-Child Chunk:

Parent Chunk (Large block, retains full context) ├── Child Chunk 1 (Small block, used for retrieval) ├── Child Chunk 2 └── Child Chunk 3

During retrieval, use the Child to search—small blocks have fine granularity and high recall; when returning, give the Parent to the LLM—the large block has complete context, allowing the LLM to understand the full picture. This balances recall rate and context integrity.

Summary

There is no silver bullet; the choice depends on the document type and scenario. The key conclusions are two: cutting too small loses context, cutting too large causes semantic dilution. In actual projects, fixed-length + sliding window is the most cost-effective starting point, with semantic chunking and Parent-Child added as needed.

Additionally, even as LLM context windows grow larger, it doesn't mean you can mindlessly stuff all retrieved chunks in. Research has found that LLMs often pay the least attention to content in the middle part of the context (the "Lost in the Middle" phenomenon)—more context does not necessarily lead to better results. Controlling the number of chunks, placing the most relevant ones at the very beginning or end, is more important than piling on volume.

Chunks are cut, Embedding has produced vectors. The next question: Between two vectors, how to compare "similarity"?

4. Similarity: How to Measure "Similarity"

Embedding turned text into vectors. The next thing to solve is: given two vectors, what metric algorithm is used to measure the degree of similarity. Three are commonly used: Cosine Similarity, Euclidean Distance, and Dot Product. Among them, Cosine Similarity is the absolute mainstream. Let's go through them one by one.

Cosine Similarity: The Most Mainstream Choice

The most commonly used is Cosine Similarity, which compares how close the directions of two vectors are:

cos(θ) = (A · B) / (|A| × |B|)

The value range is [-1, 1], where 1 means completely identical direction, 0 means orthogonal (unrelated), and -1 means completely opposite. In practice, the cosine similarity of vectors produced by Embedding models usually falls in the [0, 1] interval; truly negative values are extremely rare.

Why Cosine Similarity and not something else? Because Embedding models are optimized using Cosine Similarity during training—using another metric for comparison is like changing the grading standard for an exam; the scores naturally won't be accurate.

Brief Comparison of Other Metrics

Euclidean Distance: Measures the straight-line distance between two points in space and is greatly affected by vector length. Two vectors with the same meaning but different lengths might have a large Euclidean distance but high Cosine Similarity. Usually not the first choice.

Dot Product: When vectors have already been L2 normalized (length is 1), the dot product equals the Cosine Similarity. If vectors are normalized, using the dot product is more efficient.

One sentence of advice: If unsure which to use, use Cosine Similarity.

A Concrete Example

Concepts can become abstract easily. Let's manually calculate with two two-dimensional vectors to make it clear. Suppose we have vectors for two pieces of text (reduced to 2D for easy viewing):

A = [3, 4] ← "The weather is really nice today" B = [3.5, 3.8] ← "It's sunny outside" C = [-1, 0] ← "What to do if the computer breaks"

Cosine Similarity: Compares direction.

cos(A, B) = (3×3.5 + 4×3.8) / (√(9+16) × √(12.25+14.44)) = (10.5 + 15.2) / (5 × 5.17) = 25.7 / 25.85 ≈ 0.99 ← Almost identical direction, similar meaning cos(A, C) = (3×(-1) + 4×0) / (5 × 1) = -3 / 5 = -0.6 ← Direction is very different, meaning is unrelated

Euclidean Distance: Compares straight-line distance.

dist(A, B) = √((3-3.5)² + (4-3.8)²) = √(0.25 + 0.04) ≈ 0.54 ← Very close dist(A, C) = √((3-(-1))² + (4-0)²) = √(16 + 16) ≈ 5.66 ← Very far

Both metrics can distinguish "similar" from "unrelated," but their focus differs—Cosine looks at direction, Euclidean looks at absolute distance. They have different capabilities and different uses.

5. HNSW: Quickly Finding the Most Similar Among Millions of Vectors

With a measurement method, one can calculate the similarity between the query vector and all stored vectors one by one, sort them, and take the Top-K. The academic name for this problem is KNN (K-Nearest Neighbors). The solutions fall into two paths:

• Exact KNN: Traverse all vectors one by one, calculate similarity for each, sort, and take Top-K. 100% accurate, but complexity is O(N×D)—N vectors, each D dimensions. Manageable for tens of thousands of data points, but starts to buckle at the million level, let alone tens of millions.

• Approximate KNN (ANN): Don't calculate one by one; use an index structure to quickly lock onto candidate regions, only performing exact calculations on a few candidate vectors. Sacrifice a bit of recall accuracy for orders of magnitude performance improvement. Mainstream implementations include HNSW, IVF, PQ, LSH, etc.

Why Brute-Force Search Doesn't Work

Exact KNN is full traversal—N vectors, each D dimensions, calculating similarity one by one. For millions of data points, a million 1536-dimensional vector operations per query—just thinking about it tells you it won't work.

A MySQL analogy: It's like SELECT ... ORDER BY score LIMIT 10 without an index—every query is a full table scan. Tolerable with little data, but it collapses starting from millions of rows. ANN is like building an index for vectors; queries use the index to quickly locate candidate rows, only performing exact calculations on the few hit rows, just like MySQL using a B+Tree to scan only a few data pages.

So in practice, the ANN approximate route is used—trading controllable recall loss for orders of magnitude query performance improvement. This idea is similar to a Bloom filter: trading a tiny false positive probability for extreme space and speed savings, a very cost-effective deal in engineering.

HNSW is the most mainstream algorithm in ANN, but it's not the only one. Besides HNSW, there are schemes like IVF (K-Means clustering + inverted index), PQ (vector compression), and LSH (Locality-Sensitive Hashing). Simply put, for selection: use HNSW for under a million, IVF for tens of millions, and consider DiskANN or sharding for hundreds of millions.

The reason HNSW is the most popular is that it achieves the best balance between accuracy and speed, and its implementation is mature with a rich ecosystem. ES, Faiss, and Milvus all support it by default.

Core Insight: Skip List × Navigable Small World Graph

HNSW is a fusion of two classic data structures.

The first inspiration comes from the Skip List—adding multiple layers of "express lanes" on top of an ordered linked list:

Upper layers have fewer nodes and larger spans; the bottom layer contains all nodes. When searching, start from the top layer, jumping across many elements in one go, narrowing the range layer by layer, reducing complexity from O(n) to O(log n).

Nodes are randomly promoted to upper layers with a certain probability, requiring no global rebalancing—the randomness automatically ensures the inter-layer distribution follows the expected exponential decay pattern.

HNSW transplants this layering idea onto a graph. But there's a key problem here: Skip lists work because the linked list is ordered; comparing sizes tells you whether to go left or right. Vectors have no concept of "size"—which is larger, vector (1, 3) or (-2, 5)? You can't compare them. Having layers alone isn't enough; the problem of "how to navigate on the graph" must also be solved.

The inter-layer connection mechanism of HNSW is completely consistent with the skip list—each node is randomly assigned to several layers with a certain probability and simultaneously exists in its assigned layer and all layers below. There are no cross-node connections between layers, only vertical connections for the same node in adjacent layers. After the upper layer's coarse positioning finds an approximate nearest neighbor, follow that node's vertical connection down to the next layer, sinking layer by layer to layer 0 for a fine search.

The second inspiration comes from the Navigable Small World graph (NSW): In each layer's graph, as long as each node is connected to its few nearest neighbors, a greedy search can quickly navigate to the vicinity of the target—no global ordering needed, only local comparison. Standing at the current node, look around at neighbors, jump to the one closer to the target, and repeat.

Combining the two ideas, layering provides "express lanes," and greedy search navigates on each layer's graph—this is the entire core of HNSW. A table comparison makes it clearer:

Key Differences Between Skip List and HNSW

What the Layered Graph Looks Like

After construction, the entire graph is a layered structure like this:

An analogy—the upper layers are like "national highways" and "expressways" on a map: only key hubs are marked, but one jump can cross most of a region. The bottom layer is the "street network," dense and capable of pinpointing a specific location. When searching, first take the highway to determine the general direction, then go down to the streets to find the precise target.

Layer distribution (taking M=16, promotion probability simplified to 1/M as an example):

Note: The HNSW paper defaults to using mL = 1/ln(M) (about 0.36 when M=16) as the promotion probability. The actual Layer 1 proportion is about 36%, making the distribution "fatter" than the table above. Here, 1/M is used for simplified deduction, aiming to intuitively show the exponential decay pattern of "the higher the layer, the fewer the nodes." The order-of-magnitude conclusion remains unchanged.

For 1 million nodes, the graph's highest layer is roughly 5–6 layers. Upper layers have very few nodes, but the edge spans are large—because upper layers have few nodes and each node must connect to M neighbors, these edges naturally span very far.

Knowing what the graph looks like, let's see how it is built step by step.

How the Graph is Built: The Write Path

HNSW is essentially a multi-layer graph. Each vector is a node on the graph, and edges connect its "neighbors." When inserting a new vector, two things must be decided: which layers it goes into and who it connects to.

Step 1: Roll the Dice to Determine Layer Height

Each node uses a random function to decide up to which layer it "grows." The probability is P(level ≥ L) = (1/M)^L, with M defaulting to 16 (continuing with the simplified model here). That is: for each layer up, the probability is divided by 16. The result is that the vast majority of nodes only stay on the bottom layer, and very few nodes can reach high layers.

level = 0 while random() < 1/M: # 1/16 chance to grow one more layer up, otherwise stop level += 1

Why use randomness instead of dividing by "importance"? Because vectors have no inherent importance—placing those close to cluster centers on upper layers would mean edge data could never be found; conversely, upper layers would lose representativeness.

Random layering doesn't need to care about any properties of the data; probability theory ensures uniform inter-layer distribution. This is the same design idea as the skip list—randomness replaces complex rebalancing logic.

Step 2: Find the Nearest Neighbors in the Existing Graph

The node hasn't been inserted yet. First, take it and perform a greedy search on the existing graph—starting from the top layer, going down layer by layer, finding the node closest to it in each layer.

Step 3: Bidirectional Edge Connection

After finding the nearest neighbors, in each layer the node belongs to, establish bidirectional edges with the M nearest neighbors.

Step 4: Pruning

After connecting edges, check—if a neighbor's edge count exceeds M, keep only the M closest edges to it and prune the excess. This prevents popular nodes from being connected to too many others, which would require checking many candidates during search and slow down efficiency.

How the Graph is Searched: The Read Path

Greedy Search: How to Move Within One Layer of the Graph

The search method within each layer is called greedy search—stand at the current node, look around at all neighbors, jump to the one closer to the target, and repeat until no neighbor is closer than the current node.

Currently at node A, target vector is Q: → Calculate dist(Q, neighbor B) = 0.3 → Calculate dist(Q, neighbor C) = 0.8 → Calculate dist(Q, neighbor D) = 0.2 ← Closest! → Jump to D, continue looking at D's neighbors → Until all neighbors are farther from Q than the current node → Stop

Layer-by-Layer Drill-Down: The Complete Search Flow

With single-layer greedy search, multi-layer search is very simple—start from the top layer, perform one greedy search per layer, passing the endpoint to the next layer as the starting point. An analogy: This is like looking up a map. First, look at the national highway network to determine the general direction (top layer, a few steps across most of the region), then zoom into provincial roads to plan a specific route (middle layer), and finally pinpoint the house number at the street level (bottom layer).

Specific Steps:

• Start from the entry node at the topmost layer, perform a greedy search, and walk to the local nearest point in this layer.

• Descend to the next layer, using the endpoint from the previous layer as the starting point, and continue greedily.

• Repeat until the bottommost layer—the bottom layer contains all nodes and has the densest edges; perform the final round of fine-grained search.

• Return the K nodes encountered along the way that are closest in distance.

Without layering, in the dense bottom-layer graph, each step can only reach a neighbor. Going from one end of the graph to the other would require many steps. With layering, use the upper layers to approach the target in a few steps, then take a few precise steps in the bottom layer—total steps reduce from O(N) to O(log N).

Won't the Greedy Search Walk into a Dead End?

No. During graph construction, each node is connected to its nearest neighbors (i.e., the "bidirectional edge connection" step mentioned earlier). This structure naturally guarantees "navigability"—starting from any node and greedily walking in a closer direction will definitely approach the target area.

The long-distance edges in the upper layers have another key role: helping the search skip over areas that "look close but are actually wrong" early on, avoiding getting stuck in local optima. Of course, HNSW is an approximate algorithm and does not guarantee hitting the global nearest neighbor—this is a common design trade-off for all ANN algorithms, trading controllable recall loss for orders of magnitude performance improvement.

Three Key Parameters

ef_search is the only parameter that can be dynamically adjusted at runtime—increase it for higher accuracy requirements, decrease it for higher speed requirements, without needing to rebuild the index.

Summary

HNSW is very powerful, but semantic search has blind spots it can't handle on its own—searching for a trace ID like "SLB-20250101-abc," the Embedding model is powerless; BM25 inverted index is the right solution. Semantic search alone is not enough; the complete chain also includes several processes like recall fusion and fine-ranking, which will be expanded upon below.

6. The Complete Retrieval Chain: From Query to Fine-Ranking

Query Rewrite: What if Users Don't Know How to Ask Questions

We've discussed a lot of retrieval technology above, but one premise has been ignored—the user must first ask a good question.

In reality, how do users ask questions:

User input: "Why did that interface break again" Real intent: What was the cause of the most recent failure of the payment service order creation interface

For the retriever, the two words "interface" and "broke" carry almost zero information—the Embedding model can only guess a general direction, and BM25 is even more clueless.

Many online projects eventually find that the Embedding is fine, the vector database is fine, and the Rerank is fine; the real problem is that users don't know how to ask questions. Garbage query in, no matter how strong the subsequent chain is, it's futile.

The solution is Query Rewrite—before retrieval, let the LLM rewrite the user's vague question into a clear query suitable for retrieval. This can combine historical conversations for anaphora resolution ("that interface" → "payment service order creation interface") or supplement business context. The rewritten question is then fed into the subsequent recall and fine-ranking processes.

Query Rewrite is often one of the highest-yield single-point optimizations in practice, and most RAG frameworks (LangChain, LlamaIndex, etc.) have this step built-in.

Metadata Filtering: The First Gate

The query is rewritten. The next step is recall. But there's another process before recall—Metadata Filtering.

Many newcomers, after reading RAG introductions, form this understanding:

Query → Embedding → Vector Search Entire Database → Return Results

But in enterprise scenarios, it's actually:

Query → Metadata Filtering → Vector Search (Filtered Subset)

For example: A multi-tenant knowledge base platform where Company A's and Company B's documents are stored in the same vector database. A user searches "refund process." If not filtered by tenant_id first, refund documents from other companies will also be retrieved.

Common filtering dimensions:

Filter first, recall second. This step seems simple, but in production systems, it's the first line of defense—if the scope isn't narrowed correctly, no matter how you fine-rank afterward, it's all wrong.

Semantic search uses Dense vectors to capture semantics, BM25 uses inverted indexes to capture keywords—one searches for meaning, the other searches for literal text.

But this is just the most common two-path setup for knowledge base scenarios. The recall layer is essentially a multi-path recall framework. Where to find the answer depends on the question type:

Recall ├─ ANN (Vector Similarity Search) ├─ BM25 (Keyword Inverted Index) ├─ Graph Recall (Entity Graph, search for relationships) ├─ SQL Recall (Structured Database, precise query) ├─ Memory Recall (User Personal Memory, dynamic preferences) ├─ Conversation Recall (Historical Dialogue Context) ├─ Web Search (Real-time Internet Information) └─ Tool Recall (Call external tools to get results)

A user asks, "How to deploy that Go project I mentioned last time?"—the answer is not in the knowledge base, but in this user's historical memory or conversation context.

Knowledge bases use vector retrieval, memory uses Memory Recall, structured data uses SQL queries—different data types require different retrieval methods. Which paths the recall layer should use and how to combine them depends on the business scenario; there is no fixed answer. Multiple retrieval paths run independently, each returning a batch of results and scores. But how to merge the scores? Can you just add them directly?

Why Not Just Add Raw Scores Directly

The results returned by two paths have completely different score meanings:

ANN path: doc_A score 0.92 (Cosine Similarity) BM25 path: doc_B score 8.7 (BM25 relevance score)

How to add 0.92 and 8.7? They are completely different rulers—one is a similarity score from 0 to 1, the other is a keyword score ranging from tens to hundreds. Adding them directly is like adding "scored 92 points" and "ran 8.7 seconds" to rank. Normalization is also troublesome—it requires calibration using the score distribution of historical queries, and the distribution changes over time.

RRF: Compete on Rank, Not Score

RRF (Reciprocal Rank Fusion) has a very simple idea—since raw scores can't be directly added, discard the scores and just look at the ranks.

How Ranks Become Scores

• Multi-path superposition: A document that ranks high in more retrieval paths should get a higher score. RRF defines the final score as the sum of rank contributions from each path—each path contributes independently, and the total score naturally adds up.

• Taking the reciprocal: The higher the rank, the greater the contribution should be. Taking the reciprocal naturally satisfies this direction—rank 1 → 1/1 = 1, rank 10 → 1/10 = 0.1, rank 100 → 1/100 = 0.01.

• Adding a smoothing constant k: 1/1 and 1/2 differ by a full factor of two. In actual retrieval, the quality of the 1st and 2nd ranked documents might be similar; the score shouldn't double just because of a one-rank difference. Adding a constant k (usually 60) to the denominator flattens the curve: 1/(60+1) ≈ 0.0164, 1/(60+2) ≈ 0.0161, keeping the difference within single-digit percentages.

What the Formula Looks Like

Summarized into a formula:

score(doc) = Σ 1/(k + rank_i)

• k = 60: Smoothing constant; larger values make the curve flatter, reducing the weight difference between top ranks.
• rank_i: The document's rank in the i-th retrieval path, starting from 1.
• For a document not appearing in a certain path, that path's contribution is 0 (equivalent to infinite rank, 1/(k+∞) → 0).

Calculating with Two Retrieval Paths

`Two-path retrieval results: ANN ranking: [doc_A #1, doc_B #2, doc_C #3] BM25 ranking: [doc_B #1, doc_C #2, doc_A #3]

doc_A total score = 1/(60+1) + 1/(60+3) = 0.0164 + 0.0159 = 0.0323 doc_B total score = 1/(60+2) + 1/(60+1) = 0.0161 + 0.0164 = 0.0325 doc_C total score = 1/(60+3) + 1/(60+2) = 0.0159 + 0.0161 = 0.0320`

ANN ranks doc_A first, BM25 ranks doc_B first. After RRF fusion, B > A > C—not biased towards any single path, only looking at the comprehensive rank. Documents appearing in only one path and not the other naturally rank lower. Implementation is about 30 lines of code.

Rerank: After Coarse Filtering, Fine-Rank Again

What Recall (ANN + BM25 + RRF) does is coarse filtering—quickly fishing out dozens of candidates from a massive document collection. But these dozens of fished-out items are only "possibly relevant"; who ranks before whom is not yet accurate enough. What Rerank does is fine-ranking—using a stronger model to re-score and re-rank these dozens of candidates, pushing the truly most relevant ones to the very top.

Why Embedding Can't Rank Accurately

Embedding uses a Bi-Encoder architecture: query and doc are encoded into vectors separately, then Cosine Similarity is calculated. The problem is—during encoding, the query and doc are unaware of each other's existence; the model can only "guess" relevance afterward based on vector distance.

For example: A user searches "How to reset VPN password." After two-path recall + RRF fusion, the Top-3 might be:

VPN Password Reset Guide Vector Similarity 0.85 VPN Client Installation Tutorial Vector Similarity 0.83 VPN Account Application Process Vector Similarity 0.82

These three documents are all about VPNs; the vectors are indeed all close to the query vector, with similarities starting at 0.82—at the topic level, they are all "relevant." But topic relevance does not equal being able to answer the user's question: "How to install the VPN client" and "How to reset the VPN password" are completely different things.

After compressing text into 1024 floating-point numbers, the Bi-Encoder can no longer distinguish this level of difference—in its eyes, these are all "VPN-related documents." So Recall can "find them back," but not necessarily "rank them accurately."

Cross-Encoder: Look Together, Not Separately

Rerank uses the Cross-Encoder architecture to remedy this defect. What's the difference from Embedding's Bi-Encoder? Let's compare using the two documents above:

Bi-Encoder (Embedding's approach):

• Encode the query "How to reset VPN password" separately → Vector Q.
• Encode doc₁ "VPN Client Installation Tutorial" separately → Vector D₁, doc₂ "VPN Password Reset Guide" separately → Vector D₂.
• Calculate cos(Q, D₁) → 0.83, cos(Q, D₂) → 0.85, determine D₂ is more relevant.

The problem is that step 1 and step 2 are unaware of each other—when encoding the query, the model doesn't know what the doc looks like; when encoding the doc, it doesn't know what the query is asking. In the end, it can only rely on calculating the distance between two independent vectors to "guess" relevance, so accuracy is limited. The advantage is that vectors can be pre-computed and stored; during a query, only the query vector needs to be calculated—fast.

Cross-Encoder (Rerank's approach):

• Pair query and doc₁ together: "How to reset VPN password [SEP] VPN Client Installation Tutorial" → Feed together to the model → 0.31.
• Pair query and doc₂ together: "How to reset VPN password [SEP] VPN Password Reset Guide" → Feed together to the model → 0.98.

The model sees both sides simultaneously, reads the complete question and document, and can judge that "although both mention VPN, client installation and password reset are not the same thing"—Bi-Encoder gave 0.83, Cross-Encoder directly drops it to 0.31. The cost is that each (query, doc) pair must be re-run through the model, cannot be pre-computed, so it's much slower and can only be used for Top-N candidates.

Rerank uses a dedicated Cross-Encoder model (like bge-reranker-v2), not an Embedding model nor a general LLM. The score comes from the model's relevance judgment of the query-doc pair. The same batch of candidates goes through the same model, so scores are naturally comparable, without worrying about scale issues like with RRF.

For the same example, after Rerank:

VPN Password Reset Guide 0.98 ← Confirmed highly relevant VPN Account Application Process 0.42 ← Originally ranked 3rd, pulled up VPN Client Installation Tutorial 0.31 ← Originally ranked 2nd, pushed down

The ranking has completely changed—Rerank has widened the true gaps, directly improving the quality of the context the LLM sees.

Engineering Practice

The pipeline is now clear:

Multi-path Recall → RRF Fusion → Take Top-N (e.g., 50 items) → Rerank Fine-ranking → Take Top-K (e.g., 5 items) → Feed to LLM

Common options: bge-reranker-v2 (open source), Cohere Rerank (API), Jina Reranker (API). The selection principle is the same as for Embedding—consider language, domain, and cost.

Recall solves "don't miss anything," Rerank solves "rank correctly." The two have different roles and are both indispensable.

7. Conclusion

Walking through this article, the core concepts of RAG retrieval are basically strung together:

Query Rewrite → Metadata Filter → Recall (ANN + BM25) → RRF Fusion → Rerank Fine-ranking → LLM Generation

What truly determines the effectiveness of RAG is often Query Rewrite + Chunking + Recall + Rerank, not just the LLM itself. Summarizing a few key conclusions:

Chunking is the foundation: The chunking strategy determines how knowledge is "packaged" and directly affects what retrieval can find. Fixed chunking, sliding window, semantic chunking, and Parent-Child each have applicable scenarios. Pay attention to Lost in the Middle—more chunks are not necessarily better.

• Embedding turns semantics into math: Text becomes vectors; semantically similar items are close in space. Dense captures semantics, Sparse/BM25 captures keywords; the two are complementary.

• Use Cosine for similarity: Embedding models are trained for Cosine Similarity; using another metric is like changing the grading standard.

• HNSW solves the speed problem: Millions of vectors cannot be calculated one by one. Using the skip list's layering idea + graph's greedy search trades controllable recall loss for orders of magnitude speed improvement. Three parameters—M, ef_construction, ef_search—control the balance between accuracy and performance.

• Query Rewrite is a hidden high-yield optimization: Users don't know how to ask questions. Letting the LLM rewrite the query before retrieval is often the single-point optimization with the highest ROI. Metadata filtering is the first line of defense in a production environment—narrow the scope first, then perform recall.

• Hybrid search is the endgame: A single path has blind spots. Semantic search handles meaning, BM25 handles literal text; the two paths complement each other. RRF fuses rankings, Rerank fine-ranks—multi-path synergy, all indispensable. The essence of retrieval: It's not a single technology, but the synergy of multiple retrieval paths + ranking fusion + fine-ranking strategies—each link has its own role, and missing any link will degrade the effect.

The above summarizes the author's main understanding of the core concepts of RAG retrieval. Feedback and corrections are welcome.

Previous Reviews

1. From "Mechanical Response" to "Service Partner": Agent Engineering Practice for Dewu's Highly Controllable Intelligent Customer Service | AICon Speech Compilation

2. Dewu Recommendation System Diagnostic Agent: From "Calling APIs" to "Thinking" | AICon Speech Compilation

3. Dewu OceanBase Implementation Practice

4. AI UITester: A New Paradigm for AI Native UI Automation Testing | Dewu Technology

5. From Wild Code to Goal-Oriented Production: Engineering Practice of Dewu's Recommendation AI Harness | AICon Speech Compilation

Text / Yang Yu

Follow Dewu Technology for weekly technical干货 (dry goods / practical content)

If you find the article helpful, feel free to comment, forward, and like~

Reprinting without permission from Dewu Technology is strictly prohibited, otherwise legal liability will be pursued according to law.