← Back to the summary

Stop LLM Hallucinations by Building a RAG Pipeline from Scratch

Have you ever had this experience: you ask a large model about a recent event or internal company knowledge, and it answers with great confidence, only for you to find out it's all made up? This is the famous 'hallucination' problem of large models — they never say 'I don't know,' they just make things up with a straight face.

Why does hallucination happen? Essentially, a large model's knowledge boundary is limited by its training dataset. Content not in the training data, such as the latest industry news, your company's internal documents, or your own notes, is naturally unknown to the large model.

How to solve this problem? Currently, the most mainstream and lowest-cost solution in the industry is RAG (Retrieval-Augmented Generation). Today, we'll go from principle to code, building a minimalist RAG system from scratch using LangChain.js, to thoroughly understand how it makes large models 'tell only the truth.'

What you will learn in this article:

Understand the essence of large model hallucination and the core principles of RAG

Understand the roles of embedding models, vector retrieval, and document chunking

Manually implement a complete RAG process in JS to grasp the underlying logic

Master rapid RAG application development with LangChain.js

1. What is RAG? The Essence in One Sentence

RAG stands for Retrieval-Augmented Generation.

Its core logic is very simple:

When a user asks a question, first retrieve the most relevant document snippets from our own private knowledge base, insert these snippets as background knowledge into the Prompt, and then hand it to the large model. The large model answers based on the reference material, so naturally it won't make things up.

Broken down into three steps:

Retrieval: Based on the user's question, recall relevant text snippets from the knowledge base.
Augmented: Splice the recalled text into the Prompt to supplement the large model with background knowledge.
Generation: The large model generates an accurate answer based on the reference material.

Simply put: Check the material first, then answer the question. This is exactly the same logic as us taking an open-book exam.

2. The Core of RAG: Why is Vector Retrieval Better than Keyword Search?

Many people ask: Can't I just use keyword search? Why bother with vectors and embeddings?

An example makes it clear:

Knowledge base text: 'Guangguang is especially good at playing football'
User question: 'What sport is Guangguang good at?'

If it's a pure keyword search, searching for the word 'sport' yields no match because the original text doesn't contain that exact word; but vector retrieval can understand that 'playing football' belongs to the semantic category of 'sport' and can hit the target accurately.

This is the advantage of semantic retrieval: it matches meaning, not literal text.

2.1 Embedding Model: Turning Text into Computable Vectors

To achieve semantic retrieval, the core tool is the Embedding Model.

Its function is to convert a piece of text, an image, or a PDF into a high-dimensional mathematical vector. The closer the semantics of the content, the closer their vectors are in space.

For example:

The vectors for 'playing football' and 'playing basketball' are very close (both are sports)
The vectors for 'playing football' and 'reading and drawing' are very far apart

With vectors, we just need to convert the user's question into a vector too, then find the nearest few snippets in the knowledge base's vector library to get the most relevant reference material.

2.2 Document Chunking: 'Slicing' the Knowledge Base

Another key issue: a whole book or a document dozens of pages long cannot be directly converted into a vector and stuffed in.

First, it's too long; embedding models have input length limits.
Second, stuffing it all in mixes in a lot of irrelevant information, actually reducing retrieval accuracy.

So we need to split long documents into small, semantically complete fragments (Chunks). This can be done by paragraph, by page, or by fixed length, ensuring each fragment has an independent and complete meaning.

2.3 Vector Database: A 'Search Engine' for Massive Vectors

If your knowledge base only has a few or dozens of documents, writing a loop to calculate similarity is enough. But if there are hundreds of thousands or millions of documents, you need a dedicated Vector Database to store vectors and provide millisecond-level similarity retrieval capabilities. Common ones include Chroma, Pinecone, Milvus, Weaviate, etc.

3. Hands-on! Writing a Minimalist RAG with LangChain.js

After explaining the principles, let's jump straight into the code and implement a runnable RAG demo using Node.js + LangChain.js.

We'll use a 'Friendship Story of Guangguang and Dongdong' as our private knowledge base to see if the large model can accurately answer questions based on this knowledge base.

3.1 Environment Setup

First, install the dependencies:

bash

npm install @langchain/core @langchain/openai dotenv

Create a .env file in the project root directory and configure the model parameters (this article demonstrates using a large model compatible with the OpenAI interface + OpenAI embedding model):

env

# Chat large model configuration
MODEL_NAME=deepseek-chat
DEEPSEEK_API_KEY=YourAPIKey
OPENAI_URL=https://api.deepseek.com/v1

# Embedding model configuration
EMBEDDINGS_MODEL_NAME=text-embedding-3-small
OpenAI_API_KEY=YourOpenAIKey

3.2 Initializing Models and Building the Knowledge Base

First, import dependencies, initialize the large model and embedding model, and build our private knowledge base.

Here, we use LangChain's Document class to encapsulate knowledge snippets. Each snippet contains the main text content pageContent and metadata metadata (used for traceability and filtering, not involved in vectorization calculations).

javascript

import 'dotenv/config';
import { ChatOpenAI } from '@langchain/openai';
import { OpenAIEmbeddings } from '@langchain/openai';
import { Document } from '@langchain/core/documents';

// 1. Initialize the chat large model
const model = new ChatOpenAI({
  modelName: process.env.MODEL_NAME,
  apiKey: process.env.DEEPSEEK_API_KEY,
  temperature: 0, // Set to 0 for more rigorous answers, reducing fabrication
  configuration: {
    baseURL: process.env.OPENAI_URL
  }
});

// 2. Initialize the embedding model
const embeddings = new OpenAIEmbeddings({
  apiKey: process.env.OpenAI_API_KEY,
  model: process.env.EMBEDDINGS_MODEL_NAME,
  configuration: {
    baseURL: process.env.OPENAI_URL
  }
});

// 3. Build the private knowledge base (7 document snippets)
const documents = [
  new Document({
    pageContent: `Guangguang is a lively and cheerful little boy. He has a pair of bright, big eyes and always wears a brilliant smile. Guangguang's favorite thing is to play with his friends. He is especially good at playing football, and every time he runs on the field, he is full of energy like a ray of sunshine.`,
    metadata: { 
      chapter: 1, 
      character: "Guangguang", 
      type: "Character Introduction", 
      mood: "Lively"
    },
  }),
  new Document({
    pageContent: `Dongdong is Guangguang's best friend. He is a quiet and smart boy. Dongdong likes reading and drawing, and his paintings are always full of imagination. Although their personalities are different, Dongdong and Guangguang have known each other since kindergarten, and they have spent countless happy times together.`,
    metadata: { 
      chapter: 2, 
      character: "Dongdong", 
      type: "Character Introduction", 
      mood: "Warm"
    },
  }),
  new Document({
    pageContent: `One day, the school was going to hold a football match. Guangguang was very excited and invited Dongdong to participate. But Dongdong had never played football before, and he was worried he would drag Guangguang down. Guangguang saw Dongdong's worry, patted him on the shoulder and said: "It's okay, let's practice together, I believe you can do it!"`,
    metadata: {
      chapter: 3,
      character: "Guangguang and Dongdong",
      type: "Friendship Plot",
      mood: "Encouraging",
    },
  }),
  new Document({
    pageContent: `In the following days, Guangguang taught Dongdong how to play football every day after school. Guangguang patiently taught Dongdong how to control the ball, pass, and shoot. Although Dongdong couldn't play well at first, he never gave up. Dongdong also repaid Guangguang in his own way; he drew a picture for Guangguang, depicting two little boys playing football together on the field.`,
    metadata: {
      chapter: 4,
      character: "Guangguang and Dongdong",
      type: "Friendship Plot",
      mood: "Mutual Help",
    },
  }),
  new Document({
    pageContent: `The day of the match finally arrived. Guangguang and Dongdong stood on the field together. Although Dongdong's skills were not yet proficient, he worked very hard, and he used his observation skills to help Guangguang find the opponent's weakness. At a critical moment, Dongdong made a beautiful pass, and Guangguang received the ball and scored! They won the match, and more importantly, their friendship became even deeper.`,
    metadata: {
      chapter: 5,
      character: "Guangguang and Dongdong",
      type: "Climax/Turning Point",
      mood: "Exciting",
    },
  }),
  new Document({
    pageContent: `From then on, Guangguang and Dongdong became the best of friends at school. Guangguang taught Dongdong sports, and Dongdong taught Guangguang to draw. They learned from each other and grew together. Whenever someone asked about their friendship, they always smiled and said: "A true friend is someone who helps each other and becomes a better person together!"`,
    metadata: {
      chapter: 6,
      character: "Guangguang and Dongdong",
      type: "Ending",
      mood: "Joyful",
    },
  }),
  new Document({
    pageContent: `Many years later, Guangguang became a professional football player, and Dongdong became an excellent illustrator. Although they took different paths, their friendship never changed. Dongdong designed the pattern on Guangguang's jersey, and Guangguang would call Dongdong after every match to share his joy. They proved that true friendship can transcend time and distance, shining forever.`,
    metadata: {
      chapter: 7,
      character: "Guangguang and Dongdong",
      type: "Epilogue",
      mood: "Warm",
    },
  }),
];

3.3 Manually Implementing the Full RAG Process (Understanding the Underlying Logic)

Let's not use packaged tools first; we'll manually walk through the complete process of 'Question Vectorization → Similarity Calculation → Recall Top Snippets → Splice Prompt → Generate Answer'.

Step 1: Batch Generate Document Vectors

javascript

// Generate vectors for all knowledge base documents
const docEmbeddings = await embeddings.embedDocuments(
  documents.map(doc => doc.pageContent)
);

Step 2: Implement Cosine Similarity Calculation

The most common way to calculate vector similarity is cosine similarity. A value closer to 1 indicates higher semantic relevance.

javascript

// Cosine similarity calculation
function cosineSimilarity(vecA, vecB) {
  let dotProduct = 0;
  let normA = 0;
  let normB = 0;
  for (let i = 0; i < vecA.length; i++) {
    dotProduct += vecA[i] * vecB[i];
    normA += vecA[i] ** 2;
    normB += vecB[i] ** 2;
  }
  return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}

Step 3: Implement the Retrieval Function

javascript

// Retrieve the most relevant Top K documents
async function retrieveRelevantDocs(question, topK = 3) {
  // 1. Convert the user question into a vector
  const queryEmbedding = await embeddings.embedQuery(question);
  
  // 2. Calculate similarity with all documents
  const similarities = documents.map((doc, index) => ({
    doc,
    score: cosineSimilarity(queryEmbedding, docEmbeddings[index])
  }));
  
  // 3. Sort by similarity in descending order, take Top K
  return similarities
    .sort((a, b) => b.score - a.score)
    .slice(0, topK)
    .map(item => item.doc);
}

Step 4: Splice the Prompt and Call the Large Model

javascript

async function ragAnswer(question) {
  // 1. Retrieve relevant documents
  const relevantDocs = await retrieveRelevantDocs(question);
  
  // 2. Splice reference material into the Prompt, explicitly instructing the model to base its answer on the material
  const context = relevantDocs.map(doc => doc.pageContent).join("\n\n");
  
  const prompt = `
Please answer the user's question based on the following reference material.
If there is no relevant information in the reference material, please answer directly with "I don't know", and absolutely do not fabricate content.

【Reference Material】
${context}

【User Question】
${question}

【Answer】
`;

  // 3. Call the large model to generate an answer
  const response = await model.invoke(prompt);
  return response.content;
}

Test the Effect

javascript

// Test 1: Semantic matching question
const answer1 = await ragAnswer("What sport is Guangguang best at?");
console.log("Answer 1:", answer1);
// Output: Guangguang is best at playing football.

// Test 2: Question not in the knowledge base
const answer2 = await ragAnswer("How are Guangguang's math grades?");
console.log("Answer 2:", answer2);
// Output: I don't know

As can be seen, with RAG added, the large model can accurately answer questions based on the knowledge base and honestly says it doesn't know when encountering content it doesn't have, without fabricating.

3.4 Rapid Development with LangChain's Packaged Tools

Manual implementation is for understanding the principles. In actual development, we can directly use LangChain's packaged vector stores and retrieval chains to greatly improve efficiency.

javascript

import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { RetrievalQAChain } from "langchain/chains";

// 1. Store documents in an in-memory vector database (automatically completes vectorization)
const vectorStore = await MemoryVectorStore.fromDocuments(
  documents,
  embeddings
);

// 2. Create a retriever, set to recall Top 3 relevant snippets
const retriever = vectorStore.asRetriever(3);

// 3. Build the RAG Q&A chain with one click
const qaChain = RetrievalQAChain.fromLLM(model, retriever, {
  returnSourceDocuments: true, // Return reference sources for traceability
});

// Test call
const result = await qaChain.call({ 
  query: "What professions did Dongdong and Guangguang end up in?" 
});

console.log("Answer:", result.text);
console.log("Reference Chapters:", result.sourceDocuments.map(d => d.metadata.chapter));

4. Advanced Optimization Directions for RAG

Getting a minimalist RAG running is easy, but to use it well in a production environment, many core issues need to be solved:

4.1 Document Splitting Strategy

Splitting is the first hurdle for RAG effectiveness. Too coarse, and there's too much irrelevant information; too fine, and context is lost. Common splitting methods:

Fixed-length splitting: Simple and crude, suitable for simply structured text
Semantic splitting: Split by paragraph, chapter, or semantic boundary, yielding better results
Recursive character splitting: LangChain's built-in RecursiveCharacterTextSplitter, most commonly used in the industry

4.2 Hybrid Retrieval + Re-ranking

Pure vector retrieval also has shortcomings, such as inaccurate recall for proper nouns and precise numerical values. A common industry solution is Hybrid Retrieval: dual-path recall using vector retrieval + keyword retrieval (BM25), then re-scoring the results with a re-ranking model (Rerank), balancing both semantic and exact matching.

4.3 Vector Database Selection

For small data volumes and demos, in-memory storage is fine; for a real launch, choose a professional vector database:

Lightweight local / small projects: Chroma, FAISS
Cloud-hosted: Pinecone, Weaviate
Enterprise self-built: Milvus, Qdrant

5. Summary

The reason RAG has become the standard solution for large model implementation is that it solves three core pain points:

Solves Hallucination: Makes the large model answer based on facts, with evidence to rely on and traceability
Private Data Access: Allows the large model to use your private data without fine-tuning the model
Low Cost, Fast Iteration: Updating the knowledge base just requires modifying documents, no need to retrain the model

Today, we walked through the complete RAG development process, from principles to underlying implementation, and then to LangChain tool encapsulation. But this is just the beginning. True production-grade RAG also involves many details like Query rewriting, multi-turn conversation memory, and multi-format document parsing (PDF/Word/Tables).

If you are developing large model applications, RAG is definitely a core skill you must master.