Programmer

RAG's Full Stack, From Transformer Math to Production Retrieval

By 嘟嘟0717 · Jul 12, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

RAG is the cheapest, most auditable way to ground an LLM in proprietary or fresh data without retraining. Getting the chunking strategy, embedding model, and vector database right determines whether a system returns precise answers or hallucinates with citations.

Summary

LLMs hallucinate because they are autoregressive probability engines trained on a fixed dataset; they cannot say "I don't know" and will fabricate answers for any knowledge outside their training data. Retrieval-Augmented Generation solves this by giving the model an external brain: a user query is vectorized, the top-K most semantically similar documents are fetched from a vector database, and those documents are stuffed into the prompt so the LLM answers from evidence instead of memory.

The guide walks through every layer of this stack. It manually derives a QKV attention calculation to show how a Transformer guesses the next token, explains why keyword search fails where cosine similarity on embeddings succeeds, and contrasts fine-tuning (expensive, opaque, suited for learning a capability) with RAG (cheap, traceable, suited for learning facts). A full LangChain code example initializes ChatOpenAI with temperature 0, sets up OpenAIEmbeddings, and builds Document objects whose pageContent is vectorized while metadata is reserved for filtering and traceability.

Document chunking gets particular attention: the core rule is never to cut a sentence in half, and strategies range from chapter-based splitting for structured texts to recursive character splitting and semantic chunking for high-quality requirements. The vector database comparison covers Chroma for learning, Milvus and Pinecone for production, and PGVector for teams that want to keep everything inside PostgreSQL.

Takeaways

— LLMs generate one token at a time by computing Q·K attention scores that represent statistical co-occurrence patterns learned during training.

— Hallucination is not a bug but a direct consequence of probabilistic generation: the model cannot refuse to answer, so it fabricates plausible text for unknown facts.

— Fine-tuning teaches a model a capability (reasoning style, format) but is expensive and opaque; RAG teaches it facts by retrieving documents at query time.

— Vector semantic search finds documents by meaning, not keywords, because embeddings place semantically similar text close together in high-dimensional space.

— Embedding models (text→vector) are small, fast, and cheap, and are completely separate from generative LLMs (text→text).

— Traditional databases like MySQL cannot index high-dimensional vectors efficiently; vector databases use ANN algorithms to trade negligible accuracy for hundredfold speed gains.

— Every document chunk must preserve a complete natural semantic unit; cutting a sentence mid-thought degrades retrieval quality.

— RAG requires temperature 0 so the LLM answers deterministically from the provided documents rather than introducing creative deviations.

— Metadata on Document objects enables filtering, source attribution, and access control without polluting the vector similarity search.

Conclusions

RAG's value is not just accuracy but auditability: every answer can be traced to a specific ingested document, which matters more in regulated industries than raw performance.

The guide's insistence on manually deriving a QKV calculation before discussing RAG reflects a practical truth: debugging retrieval failures often requires understanding why the model attended to the wrong context.

Temperature 0 is non-negotiable for RAG, yet many tutorials omit this detail, leading to systems that still hallucinate despite having the correct documents in the prompt.

Choosing between PGVector and a dedicated vector database is really a choice about operational complexity versus architectural simplicity; the right answer depends on whether your team already runs PostgreSQL in production.

Concepts & terms

QKV Self-Attention

The mechanism inside a Transformer where each token emits a Query (what it's looking for), a Key (its identity), and a Value (its content). The dot product of Q and K determines attention weights, which are used to compute a weighted sum of V values, fusing context into each token's representation.

Cosine Similarity

A measure of the angle between two vectors, ranging from -1 to 1. In RAG, it quantifies semantic similarity between a query embedding and document embeddings; a score near 1 means the texts are semantically close regardless of wording.

ANN (Approximate Nearest Neighbor)

A class of algorithms that find vectors close to a query vector in high-dimensional space without exhaustively comparing every stored vector. ANN trades a tiny amount of accuracy for orders-of-magnitude faster retrieval, making real-time semantic search feasible.

Embedding Model

A specialized model that converts unstructured data (text, images, audio) into fixed-length numerical vectors. Unlike generative LLMs, embedding models are small, fast, and cheap; they produce semantic coordinates rather than natural language.

Temperature

A parameter controlling the randomness of an LLM's output, typically ranging from 0 to 2. At 0, outputs are deterministic and strictly follow the input prompt; at higher values, the model introduces creative variation. RAG systems must use temperature 0 to prevent fabricating details not present in the retrieved documents.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗