RAG Stops LLM Hallucination by Forcing a Lookup Before Every Answer
Written at the beginning: Today I learned about RAG (Retrieval-Augmented Generation), and the teacher said something that was a real eye-opener — "A large model won't say 'I don't know'; it will just make up an answer." If you ask it who won the World Cup yesterday, it doesn't know (its knowledge is cut off at its training time), but it won't say "I don't know." It will fabricate an answer for you. And it will sound quite plausible — this is the legendary "AI hallucination." RAG is the solution to this problem: don't let your AI make things up; make it look up information first, and only answer after it has found it.
1. AI's "Pretending to Know" — Where Does Hallucination Come From?
1.1 AI's Knowledge Isn't "Live"
The teacher said:
"The knowledge a large model has depends on the dataset it was given during training. If you ask it about recent events, or your company's internal private documents, the LLM won't know."
But the problem is — an LLM won't say "I don't know."
It will, based on its limited training knowledge, forcefully fabricate an answer that sounds reasonable. This is the so-called Hallucination.
You ask it: "How many gold medals did the Chinese team win at the 2024 Paris Olympics?" If its training data was cut off in 2023, it doesn't know. But it won't say "I don't know." It will say: "The Chinese team won 38 gold medals..." — which is completely made up.
1.2 Three Solutions to Hallucination
The teacher said:
"How to solve the hallucination problem in large models? Think about it."
| Solution | Cost | Description |
|---|---|---|
| Fine-tuning | High | Used by large companies and specialized fields; requires computing power and data |
| RAG (Retrieval-Augmented Generation) | Low | Look up a knowledge base first, then answer |
| Re-training | Very High | Almost no one re-trains a large model just for new knowledge |
RAG is the most cost-effective solution.
2. RAG Isn't Magic, It's "Look It Up First, Then Answer"
2.1 What is RAG?
The teacher said:
"RAG: Retrieval — Augmented — Generation."
User asks a question
↓
【Retrieval】Search a knowledge base for relevant document fragments
↓
【Augmented】Insert the found content into the Prompt
↓
【Generation】The LLM answers the question based on the augmented context
Core idea: Before sending a Prompt to the LLM, first look it up in a knowledge base, insert the relevant documents as background knowledge, and then let the LLM answer.
2.2 Why Not Use Keyword Search?
The teacher said:
"Keyword search — text matching — isn't very good. Vector semantic queries are much better."
For example:
| Keyword | Searching with LIKE | Searching with Vectors |
|---|---|---|
| "Guangguang" | Only matches paragraphs containing "Guangguang" | Even searching for "friend" can find paragraphs related to Guangguang |
| "football match" | Only matches paragraphs containing "football match" | Searching for "sports" can also find football-related content |
Vector search finds "meaning"; keyword search finds "characters."
2.3 The "Mathematical Magic" of Vectors
The teacher drew a very intuitive example:
Fruit vector: [0.9, 0.3]
Apple vector: [0.9, 0.5] ← very close to fruit
Banana vector: [0.9, 0.1] ← also quite close to fruit
Stone vector: [0.1, 0.9] ← very far from fruit
Every word can be represented as a multi-dimensional vector. Words with similar semantics have a small angle between them in vector space (high cosine similarity). Semantically unrelated words have a large angle.
Therefore, RAG generally uses an Embedding Model for semantic search, which is much more accurate than keyword search.
3. Implementing RAG with LangChain: A Complete Story
3.1 Preparing the Knowledge Base Documents
Looking at index.mjs, the teacher used a little story about "Guangguang and Dongdong":
const documents = [
new Document({
pageContent: `Guangguang is a lively and cheerful little boy. He has a pair of bright, big eyes and always wears a brilliant smile.
Guangguang's favorite thing is to play with his friends, and he is especially good at playing football...`,
metadata: {
chapter: 1,
character: "Guangguang",
type: "Character Introduction",
mood: "Lively"
},
}),
new Document({
pageContent: `Dongdong is Guangguang's best friend. He is a quiet and intelligent boy.
Dongdong likes reading and drawing, and his drawings are always full of imagination...`,
metadata: {
chapter: 2,
character: "Dongdong",
type: "Character Introduction",
mood: "Warm"
},
}),
// ...7 fragments in total
];
Document is the smallest unit for Embedding in LangChain:
| Field | Purpose | Analogy |
|---|---|---|
| pageContent | The text content to be vectorized | A paragraph |
| metadata | Additional information, not involved in vectorization | Tags (chapter, character, type) |
3.2 Vectorizing Documents and Storing in an In-Memory Vector Store
const embeddings = new OpenAIEmbeddings({
model: process.env.EMDEDDING_MODEL_NAME,
});
const vectorStore = await MemoryVectorStore
.fromDocuments(documents, embeddings);
console.log(`Vector store created successfully, ${vectorStore.memoryVectors.length} documents have been vectorized`);
MemoryVectorStore is an in-memory vector storage provided by LangChain. It converts documents into vectors via an Embedding model and stores them in memory.
The teacher said:
"For simple cases, put it in memory; for complex ones, put it in a vector database (like PostgreSQL + pgvector)."
3.3 Creating a Retriever
const retriever = vectorStore.asRetriever({
k: 3 // Return the top 3 most similar results
});
A Retriever is a standard entry point:
Knowledge Base → Documents → Document → Embedding → MemoryVectorStore → Retriever
3.4 Querying and Getting Relevant Documents
const question = "How did Dongdong and Guangguang become friends?";
const docs = await retriever.invoke(question);
const scoredResults =
await vectorStore.similaritySearchWithScore(question, 3);
The output includes similarity scores:
[Document 1] Similarity: 0.9234
Content: Dongdong is Guangguang's best friend, he is a quiet and intelligent boy...
Metadata: chapter=2, character=Dongdong, type=Character Introduction
[Document 2] Similarity: 0.8876
Content: One day, the school was going to hold a football match...
Metadata: chapter=3, characters=Guangguang and Dongdong, type=Friendship Plot
[Document 3] Similarity: 0.7543
Content: In the following days, Guangguang would teach Dongdong to play football every day after school...
Metadata: chapter=4, characters=Guangguang and Dongdong, type=Friendship Plot
The closer the similarity is to 1, the more relevant it is to the question.
3.5 Augmenting the Prompt and Generating an Answer
const context = docs
.map((doc, i) => `[Fragment ${i}]\n ${doc.pageContent}`)
.join("\n\n-----\n\n");
const prompt = `You are a teacher who tells friendship stories.
Answer the question based on the following story fragments, using warm and vivid language.
Story fragments:
${context}
Question: ${question}
The teacher's answer:`;
const response = await model.invoke(prompt);
The retrieved document fragments are inserted into the "Story fragments" part of the Prompt. The LLM answers the question based on this augmented context.
4. A Review of the Complete RAG Flow
User asks: "How did Dongdong and Guangguang become friends?"
↓ 1. Retrieval
Convert the question to a vector → Search in the vector store → Find the 3 most similar fragments
↓ 2. Augmented
Assemble the 3 fragments into the "Story fragments" part of the Prompt
↓ 3. Generation
The LLM generates an answer based on the story fragments + the question
↓ Output
"Dongdong and Guangguang have been good friends since kindergarten...
Guangguang invited Dongdong to join a football match. Although Dongdong couldn't play,
Guangguang patiently taught him, and their friendship grew deeper because of it..."
Without RAG, the LLM can only answer based on its own training knowledge. But if this story wasn't in its training data, the LLM would "make it up."
With RAG, the LLM first checks a knowledge base and answers based on real content — hallucination is greatly reduced.
5. RAG Architecture in a Real Project
The teacher said:
"To implement semantic queries, RAG needs to be based on vectors. Documents are vectorized and stored in a vector database. When querying, the Prompt is also vectorized, and a similarity search is performed in the database to find semantically similar text blocks."
RAG architecture in a production environment:
Raw Documents (PDF, Word, web pages, etc.)
↓ Text extraction + Chunking
Text Fragments
↓ Embedding Model
Vectors
↓ Storage
Vector Database (PostgreSQL+pgvector, Pinecone, Weaviate)
↓ Create
Retriever
↓ Query
User Question → Vectorization → Similarity Search → Relevant Fragments → Augmented Prompt → LLM Generation
| Component | Purpose | What We Used |
|---|---|---|
| Document Chunking | Splits long documents into semantic paragraphs | Manually split into 7 fragments |
| Embedding Model | Converts text into vectors | OpenAIEmbeddings |
| Vector Storage | Stores vectors, supports similarity search | MemoryVectorStore |
| Retriever | Standard entry point, input a question, output documents | vectorStore.asRetriever() |
6. Summary: RAG is an LLM's "External Knowledge Base"
| Concept | Description |
|---|---|
| LLM Hallucination | An LLM doesn't know and won't say it doesn't know; it will forcefully make up an answer |
| RAG | Retrieval Augmented Generation |
| Retrieval | Finding document fragments from a knowledge base that are relevant to the question |
| Augmented | Adding the fragments into the Prompt context |
| Generation | The LLM answers based on the augmented context |
| Vector Search | Using Embeddings to find semantically similar text |
| Document | The smallest unit for Embedding in LangChain |
| MemoryVectorStore | In-memory vector storage, a lightweight solution |
| Retriever | retriever.invoke(question) standard entry point |
The core idea of RAG: Don't let your AI make up answers out of thin air. Give it materials to look up first, and only let it answer after it has found them.
Written at the End
The biggest takeaway today was thoroughly understanding the principle of RAG. Before, I thought "AI making up answers" was some kind of metaphysical problem. Now I know — an LLM isn't intentionally deceiving you; it genuinely doesn't know, but it's "too embarrassed" to say it doesn't know. RAG is like giving the LLM a "secretary" — when it encounters something it doesn't know, it first looks up the information, and only speaks after finding it.
Next time an interviewer asks you: "What is the working principle of RAG? How do you implement it with LangChain?"
You can calmly say:
"RAG stands for Retrieval Augmented Generation. The core idea is: before sending a Prompt to the LLM, first retrieve relevant documents from a knowledge base. The process has three steps: ① Retrieval — convert the user's question into a vector using an Embedding model, perform a similarity search in a vector store, and find the top N most relevant document fragments; ② Augmented — add the retrieved fragments as context into the Prompt; ③ Generation — the LLM answers the question based on the augmented Prompt. In LangChain, you define knowledge fragments using the Document object, create a vector store with MemoryVectorStore.fromDocuments(), create a retriever with vectorStore.asRetriever(k), and call retriever.invoke(question) to retrieve relevant documents. This approach effectively solves the hallucination problem in LLMs, allowing the AI to answer based on real knowledge rather than fabricating it out of thin air."
Then, looking at the interviewer's satisfied expression, silently think to yourself: Nailed it again.
All code examples in this article are from classroom learning materials and are real and runnable.