RAG's Full Stack, From Transformer Math to Production Retrieval

RAG From Beginner to Practice: A Systematic Learning and Review Guide

This article is compiled from RAG study notes, covering the full-chain knowledge system from LLM fundamentals to complete RAG practice, suitable as a systematic review reference.

1. Prerequisites: How LLMs Work

Before understanding RAG, you need to figure out how large models actually "speak."

1.1 Core Logic: Look at the Above, Guess the Below

The essence of an LLM is an autoregressive language model—it does not output an entire paragraph at once, but rather spits it out one word at a time.

Input: "I am very"
  ↓ Model processing
Output probabilities: "happy" 62%, "sad" 18%, "tired" 8%, others...
  ↓ Select the highest
Output: "happy" → Concatenated into "I am very happy"
  ↓ Feed it back in, guess the next word...

Each time it only outputs one more token (≈ one word or character). You think it's having a fluent conversation, but it's actually playing the "guess the next word" game thousands of times.

1.2 The Core Guessing Mechanism: QKV Self-Attention

In the Transformer architecture, how does each token know which preceding words to focus on? It relies on the QKV self-attention mechanism.

Concept	Full Name	Plain Meaning	One-Liner
Q	Query	"Who am I looking for?"	The "missing person notice" issued by the current token
K	Key	"Who am I?"	The identity label each token gives itself
V	Value	"What do I have?"	The actual information carried by each token

1.3 Analogy—Looking Up Materials in a Library

You need to research "Transformer attention mechanism" for a paper:

Q: The research question in your mind—"How exactly is attention calculated?"
K: The title, tags, and classification number of each book—helps you locate the book
V: The actual content in the book—what you really need to read

1.4 Complete Calculation Process (Example)

Input "I love eating ap", the model needs to guess the next word. Take the last token "ap" as an example (it is responsible for guessing the next word):

Step 1: Each word generates its own Q, K, V through trained W_Q, W_K, W_V
(Matrix multiplication, results shown directly here)

  token      Q               K               V
  ─────────────────────────────────────────────────
  "I"     [0.2, 0.5, 0.1]  [0.3, 0.1, 0.8]  [1.0, 0.2, 0.5]
  "love"  [0.4, 0.1, 0.6]  [0.5, 0.7, 0.2]  [0.1, 0.9, 0.3]
  "eating"[0.7, 0.3, 0.4]  [0.6, 0.2, 0.5]  [0.4, 0.1, 1.0]
  "ap"    [0.6, 0.2, 0.7]  [0.4, 0.5, 0.3]  [0.3, 0.4, 0.8]

Step 2:
  Q("ap") · K("eating") = 0.92  ← Highest! The verb "eating" is the strongest signal for guessing food
  Q("ap") · K("I") = 0.76
  Q("ap") · K("ap") = 0.55
  Q("ap") · K("love") = 0.58

Why does "eating" score the highest? Because "eating apple" and "eating banana" appear frequently in the training data, gradient descent makes the vector directions of W_K("eating") and W_Q(food-related tokens) increasingly aligned. The Q·K score is a statistical regularity learned from training.

Step 3: softmax converts scores into weights (summing to 1)

  Raw scores: [0.92,  0.76,  0.58,  0.55]
  softmax  → [0.31,  0.26,  0.22,  0.21]

  "eating" 31% | "I" 26% | "love" 22% | "ap" 21%

Step 4: Weighted average of all words' V using the weights, fusing the entire sentence's information

  Output = 0.31 × V("eating") + 0.26 × V("I") + 0.22 × V("love") + 0.21 × V("ap")
       = 0.31×[0.4, 0.1, 1.0]
       + 0.26×[1.0, 0.2, 0.5]
       + 0.22×[0.1, 0.9, 0.3]
       + 0.21×[0.3, 0.4, 0.8]
       ─────────────────────────
       = [0.47, 0.37, 0.67]    ← A vector fusing information from the entire text

Step 5: This vector goes through a fully connected layer + softmax → vocabulary probabilities

  "apple" → 62%   ← ✅ Highest!
  "fruit" → 18%
  "bread" → 12%
  "banana"→  5%
  others  →  3%

  Model output: "ple", combined with the previous "ap" to form the complete "apple"

2. The Achilles' Heel of LLMs: Where Hallucinations Come From

2.1 What is Hallucination

The knowledge a large model knows depends entirely on the dataset it was fed during training. There are two blind spots:

Knowledge has a cutoff date: It doesn't know what happened after the training data.
Hasn't seen private data: Internal corporate documents, product manuals, and specifications are also unseen.

The key problem is: LLMs won't say "I don't know." As a probabilistic generation model, when faced with an unknown question, it won't refuse to answer but will "fabricate" a plausible-sounding but factually incorrect answer based on learned language patterns—this is hallucination.

2.2 Scenario Hazards of Hallucination

Scenario	Hallucination Manifestation	Consequence
Medical Q&A	Fabricates non-existent treatment plans	Life-threatening
Legal Consultation	Cites fictional laws and precedents	Legal disputes
Enterprise Customer Service	Gives incorrect return/refund policies	Customer complaints, refund losses
Financial Analysis	Fabricates financial report data	Investment decision errors

3. Two Paths to Solving Hallucination: Fine-tuning vs RAG

3.1 Fine-tuning

Continues training the large model with new data, allowing the model to "learn" new knowledge.

Disadvantages:

Problem	Description
High Cost	Requires large amounts of labeled data + GPU compute
Long Cycle	Training takes days to weeks
Untraceable	Remembers the knowledge but can't say where it learned it from
Hard to Update	Knowledge expires → retrain

Applicable Scenarios: Large companies, specialized vertical domains (medical image diagnosis, legal document generation)—where the model needs to "learn a capability," not just "memorize a bunch of facts."

3.2 RAG (Retrieval-Augmented Generation)

Does not train the model, but equips it with an external knowledge base. For each query, it first searches for materials, stuffs the materials into the Prompt, and then has the model answer based on the materials.

Traditional LLM:
  User asks → LLM answers from training memory → Possible hallucination ❌

RAG:
  User asks → Knowledge base retrieval → Find relevant documents → Insert into Prompt → LLM answers based on materials ✅

3.3 Comparison Summary

Dimension	Fine-tuning	RAG
Cost	High (GPU + Labeling + Time)	Low (Only Embedding + Storage)
Knowledge Update	Retrain	Add/delete documents
Explainability	Black box	Traceable to specific documents
Applicable Direction	Learn "capability" (reasoning style, format)	Learn "knowledge" (facts, norms, policies)
Hallucination Control	Not guaranteed, may learn incorrect info	Significantly reduced (evidence-based)

4. Deep Dive into RAG Core Mechanisms

4.1 Meaning of the Three Letters

Letter	Full Name	Meaning	Plain Understanding
R	Retrieval	Retrieval	Search the knowledge base for relevant documents
A	Augmented	Augmentation	Stuff the found documents into the Prompt
G	Generation	Generation	LLM generates an answer based on the augmented Prompt

4.2 Retrieval Phase: How to Find Relevant Documents?

Option 1: Keyword Search (Traditional approach, poor results)

Directly perform string matching within documents.

User asks: "What is that red, crunchy fruit?"
Document contains: "Red Fuji apple is a..."

Keyword matching: "red"✗, "crunchy"✗, "fruit"✗ → Not found ❌

Fundamental Flaw: Only does literal matching, doesn't understand semantics. A different phrasing or synonym substitution causes a miss.

Option 2: Vector Semantic Search (Standard for modern RAG)

Converts text into vectors (a string of numbers), using the angle between vectors to measure semantic similarity.

Intuitive understanding with two dimensions (real models have 768~4096 dimensions):

         Edibility      Hardness
Fruit →  [0.9,          0.3]
Apple →  [0.9,          0.5]
Banana → [0.8,          0.3]
Stone →  [0.1,          0.9]

Fruit vs Apple: Angle ≈ 10°  → Strongly related ✅
Fruit vs Banana: Angle ≈ 12° → Strongly related ✅
Fruit vs Stone: Angle ≈ 75°  → Basically unrelated ❌

Even if a user asks "red, crunchy fruit," its vector will have a small angle with the vector for "Red Fuji apple"—it searches for meaning, not literal text.

5. Embedding Models and Vectorization

5.1 What is an Embedding Model

A model specifically designed to convert text (images/speech/PDFs) into vectors. It's a different thing from a generative LLM:

Comparison Dimension	Embedding Model	Generative LLM
Task	Text → Vector	Text → Text
Model Size	Small (hundreds of MB)	Large (hundreds of GB)
Inference Cost	Low (cheap)	High (expensive)
Inference Speed	Fast	Slow
Representative Models	text-embedding-3-small	GPT-4o, Claude

5.2 The Significance of Vectorization

Turning unstructured text into numbers a computer can calculate:

"Hello there" → [0.21, -0.53, 0.78, ..., 0.09]  (768 dimensions)
"今日は" → [0.02, -0.61, 0.81, ..., 0.12]  (Japanese for "Hello", similar meaning, vectors are also close)

The semantic similarity of two vectors is calculated using Cosine Similarity, ranging from -1 to 1, where closer to 1 means more similar.

5.3 Supported Data Types

Not just text—as long as the embedding model supports it, anything can be vectorized for semantic search: text, PDFs, images (multimodal embedding models), speech (vectorized after transcription), code files.

6. Vector Databases

6.1 Why a Specialized Vector Database is Needed

Traditional databases (MySQL, PostgreSQL) store structured data (numbers, strings), with indexes based on B-Tree / Hash, excelling at exact matching. But similarity calculation for high-dimensional vectors requires ANN (Approximate Nearest Neighbor) indexes, which traditional databases cannot do.

Vector databases are specifically designed for this, with a core trifecta:

Capability	Description
Store	Persistent storage of massive high-dimensional vectors
Search	Millisecond-level ANN (Approximate Nearest Neighbor) retrieval
Filter	Scalar filtering combined with metadata ("only search documents from Chapter 3")

6.2 Common Vector Databases

Product	Positioning	Features
Chroma	Lightweight Open Source	Suitable for learning and small projects, dual Python/JS SDK
Milvus	Enterprise-grade Open Source	Distributed, high-performance, multiple index types
Pinecone	Cloud Service	Fully managed, ready out-of-the-box
Weaviate	Open Source	Built-in vectorization module, supports hybrid search
Qdrant	Open Source	Written in Rust, excellent performance
PGVector	PostgreSQL Plugin	Store vectors directly in PG, reducing architectural complexity

6.3 Similarity Retrieval Process

1. User asks: "How did Guangguang and Dongdong become friends?"
       ↓
2. Embedding model vectorizes the question → [0.34, -0.12, 0.78, ...]
       ↓
3. Vector database uses ANN algorithm to find the most similar Top-K document vectors
       ↓
4. Returns the K documents with the highest similarity (e.g., K=3)

7. Document Chunking Strategies

7.1 Why Chunking is Necessary

Two reasons:

Limited Context Window: An entire book cannot fit into a Prompt.
Retrieval Precision: For a 5000-word document, if a user asks about a small detail within it, irrelevant content will dilute the vector similarity.

7.2 Comparison of Chunking Strategies

Strategy	Applicable Scenarios	Advantages	Disadvantages
By Chapter	Highly structured long texts	Semantically complete	Granularity may be too coarse
By Paragraph	General articles	Natural semantic boundaries	Uneven paragraph lengths
By Fixed Character Count	General purpose	Simple and controllable	May cut off semantics
Recursive Character Splitting	LangChain default	Splits progressively by delimiter priority	Requires tuning chunk_size
Semantic Chunking	High-quality requirements	Uses model to judge natural breakpoints	High cost

7.3 Core Principle

Each chunk must maintain complete natural semantics; never cut a sentence in half.

7.4 Actual Chunking in hello-rag

The story was chunked by chapter into 7 Documents, each a complete plot segment:

Chapter 1: Character Introduction (Guangguang)
Chapter 2: Character Introduction (Dongdong)
Chapter 3: Friendship Plot (Guangguang invites Dongdong to play soccer)
Chapter 4: Friendship Plot (Helping each other practice)
Chapter 5: Climax and Turning Point (Winning the match)
Chapter 6: Ending (Friendship deepens)
Chapter 7: Epilogue (Many years later)

8.1 Environment Configuration

import 'dotenv/config'

Automatically reads .env, loading sensitive information into process.env:

OPENAI_API_KEY=sk-xxxxxxxx
OPENAI_API_BASE_URL=https://your-proxy.com/v1
MODEL_NAME=gpt-4o
EMBEDDING_MODEL_NAME=text-embedding-3-small

8.2 Initializing the LLM (Generative Model)

import { ChatOpenAI } from 'langchain/openai'

const model = new ChatOpenAI({
  temperature: 0,                          // Must be 0 for RAG scenarios
  model: process.env.MODEL_NAME,           // e.g., gpt-4o
  apiKey: process.env.OPENAI_API_KEY,
  configuration: {
    baseURL: process.env.OPENAI_API_BASE_URL,  // Proxy/relay address
  }
})

temperature parameter (OpenAI range 0~2):

Value	Effect	RAG Suitability
0	Responses nearly identical each time, strictly follows input	✅ Required for RAG
0.7~1	Some creativity	❌ May deviate from documents
>1.5	Highly creative	❌ Prone to fabrication

8.3 Initializing the Embedding Model

import { OpenAIEmbeddings } from 'langchain/openai'

const embeddings = new OpenAIEmbeddings({
  apiKey: process.env.OPENAI_API_KEY,
  model: process.env.EMBEDDING_MODEL_NAME,      // e.g., text-embedding-3-small
  configuration: {
    baseURL: process.env.OPENAI_API_BASE_URL,
  }
})

Key Distinction: ChatOpenAI (LLM, responsible for generating answers) and OpenAIEmbeddings (Embedding model, responsible for text→vector) are two independent instances, each with its own role.

8.4 Building the Document Array

import { Document } from 'langchain/core/documents'

const document = [
  new Document({
    pageContent: `Guangguang is a lively and cheerful little boy...`,  // Body text, participates in vectorization
    metadata: {                                     // Tags, do not participate in vectorization
      chapter: 1,
      character: "Guangguang",
      type: "Character Introduction",
      mood: "Lively"
    },
  }),
  // ... 7 Documents in total
]

The two fields of a Document:

Field	Role	Participates in Vectorization?	Use Case
`pageContent`	Document body	✅ Yes	Main body for retrieval matching
`metadata`	Additional tags	❌ No	Filtering, traceability, access control

Three practical uses for metadata:

Filtering: After searching, keep only results where type === "Character Introduction"
Traceability: Attach "Referenced from Chapter 3" to the answer
Permissions: Filter by document classification level

8.5 Defining the Query

const question = [
  "What is the friendship between Guangguang and Dongdong like?"
]

Subsequent process: Vectorize question → Similarity retrieval → Return most relevant Document → Concatenate pageContent into Prompt → LLM generates answer.

The code up to this point is still in the "preparation phase"; the actual retrieval + augmentation + generation logic needs to be completed later.

9. Complete RAG Process Summary

═══════════════════════════════════════════════════════════════
                        RAG Complete Data Flow
═══════════════════════════════════════════════════════════════

【Offline Phase: Knowledge Ingestion  —  Done once when documents update】
┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│ 1.Load   │ → │ 2.Chunk  │ → │ 3.Embed  │ → │ 4.Store  │
│ Document │    │          │    │          │    │ VectorDB │
└──────────┘    └──────────┘    └──────────┘    └──────────┘

【Online Phase: Q&A  —  Executed for every user query】
┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│ 5.User   │ → │ 6.Embed  │ → │ 7.Similarity│ → │ 8.Insert │
│ Question │    │ Question │    │ Retrieve K │    │ Prompt   │
└──────────┘    └──────────┘    └──────────┘    └─────┬────┘
                                                       ↓
                                                ┌──────────┐
                                                │9.LLM Gen │
                                                │  Answer  │
                                                └──────────┘
═══════════════════════════════════════════════════════════════

Summary of the Three Key Steps:

Step	What it does	Core Tools
R Retrieval	Vectorize question → Vector DB similarity calculation → Return Top-K documents	Embedding Model + VectorDB
A Augmented	Concatenate retrieved documents into the Prompt as background knowledge	Prompt Template
G Generation	LLM reads reference materials, generates an evidence-based answer	ChatOpenAI / LLM

10. How to Systematically Learn and Review RAG

10.1 Seven-Layer Progressive Learning Path

Layer 1: LLM Fundamentals
  ├── Autoregressive generation (token level, one word at a time)
  ├── Transformer Attention (QKV calculation process, manually derive it once)
  └── LLM Limitations (hallucination, knowledge cutoff, limited context window)

Layer 2: Why RAG
  ├── Root cause of hallucination (probabilistic generation + knowledge blind spots)
  ├── Fine-tuning vs RAG comparison (cost, explainability, update method)
  └── Core idea of RAG: Retrieval → Augmentation → Generation

Layer 3: Embeddings and Vectors
  ├── Embedding model vs Generative LLM (different tasks, different costs)
  ├── Semantic meaning of vectors (multi-dimensional numbers = semantic coordinates)
  ├── Cosine similarity calculation principle
  └── Multimodal vectorization (text, images, speech)

Layer 4: Vector Databases
  ├── Why MySQL can't do it (B-Tree cannot index high-dimensional vectors)
  ├── ANN Approximate Nearest Neighbor (sacrifice tiny precision for hundredfold speed)
  ├── Mainstream selection (Chroma for entry → Milvus/Pinecone for production)
  └── Hybrid search (vector similarity + metadata scalar filtering)

Layer 5: Document Processing
  ├── Chunking strategies (by chapter/paragraph/fixed count/semantic/recursive)
  ├── Chunk Size and Overlap tuning
  ├── Metadata design (serving filtering and traceability)
  └── Multi-format loading (PDF, web pages, databases, APIs)

10.2 Core Concept Quick Reference Table

Concept	One-Liner Explanation	Keywords
LLM Hallucination	Model doesn't know but won't refuse, fabricates answers based on probability	Knowledge cutoff, probabilistic generation
RAG	Search materials first then answer, giving LLM an "external brain"	Retrieval + Augmented + Generation
Fine-tuning	Continue training the model to learn new knowledge	High cost, GPU, labeled data
Embedding Model	A dedicated model for text→vector	Cheap, fast, small, different from LLM
Vector	The "semantic coordinates" of a piece of text	Multi-dimensional array, cosine similarity
Cosine Similarity	Cosine of the angle between two vectors, measures semantic relevance	-1~1, closer to 1 is more similar
Vector Database	Stores vectors + fast ANN retrieval	Chroma, Milvus, Pinecone
Chunk	A text fragment after splitting	Maintain complete semantics
Document	`pageContent` (body) + `metadata` (tags)	Body is vectorized, tags are not
metadata	Tag attributes of a document	Filtering, traceability, permissions
temperature	Randomness of LLM output	Set to 0 for RAG
Top-K	Retrieve the K most similar items	Usually 3~10
QKV	Core mechanism of Transformer Attention	Q finds, K labels, V gives
LangChain	AI application development framework	Wraps LLMs, vector DBs, document operations

10.4 Self-Check Checklist

What is LLM hallucination? Why does it happen?
What are the pros and cons of fine-tuning and RAG? What scenarios are they suitable for?
What do the three letters of RAG stand for? What does each step specifically do?
Why is keyword search worse than vector semantic search?
What is the difference between an embedding model and an LLM?
What problem does a vector database solve that MySQL cannot?
What are pageContent and metadata in a Document used for?
What is the core principle of document chunking?
What is the complete RAG chain from user query to LLM answer?
What do Q, K, and V stand for? How do they cooperate in Attention?
How is cosine similarity calculated? Why use cosine instead of Euclidean distance?
What should temperature be set to in a RAG scenario? Why?

This article is written based on practical code using LangChain + OpenAI Embeddings, covering everything from the underlying QKV mechanism of LLMs to the complete RAG engineering chain. It is recommended to bookmark as a systematic review manual.