Backend · Artificial Intelligence · Agent

A RAG Pipeline from Scratch: Chunking, Indexing, Retrieval, Reranking, and Generation

By IvanCodes · Aug 10, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Most RAG tutorials stop at a LangChain wrapper. This one forces you to see every handoff — chunker, embedder, retriever, reranker, prompt — so you can debug a wrong answer without guessing which stage broke it.

Summary

Dumping entire documents into a large language model breaks down fast: context windows fill up, costs spike, and the model still misses answers buried in noise. A proper RAG pipeline solves this by splitting documents into focused chunks, converting them into vectors, and retrieving only the most relevant fragments at query time. This walkthrough builds that pipeline step by step, from a paragraph-aware chunker with overlap to a cosine-similarity retriever and an optional BGE Reranker that re-scores candidates before generation.

Each stage carries its own failure modes. Chunk boundaries that cut a condition from its conclusion, embedding models that don't match the domain, and exact-match blind spots for error codes or version numbers all degrade results silently. The piece maps these failure modes to a systematic debugging order: check the source documents first, then chunk integrity, then retrieval recall, then ranking, and only then the generation model and prompt.

The full Python demo reads local TXT files, builds a JSON index with OpenAI embeddings, and answers questions with inline citations. It's a teaching tool, not production software, and the final section catalogs what's missing: real-time data access, permission filtering, prompt-injection defenses, and the jump to Agentic RAG when the base retrieval is stable.

Takeaways

— Chunking directly sets the ceiling on retrieval quality; a chunk that mixes unrelated topics dilutes its vector and won't match specific queries well.

— Overlap (10–20%) prevents answers from being split across chunk boundaries, but too much overlap creates duplicate chunks that waste index space.

— Embedding models for documents and queries must be identical, or similarity scores become meaningless because the vectors live in different spaces.

— Vector retrieval alone misses exact matches like error codes and version numbers; combining it with BM25 or keyword search closes that gap.

— A Reranker re-scores retrieved candidates with a cross-encoder that sees the query and document together, but it can't rescue an answer that was never recalled.

— Debugging a wrong RAG answer follows a strict upstream order: source documents → chunk completeness → retrieval recall → ranking → prompt and model.

— Metadata such as source, section, and permissions should be stored alongside each chunk to enable citation display and access control at query time.

Conclusions

RAG pipeline quality is bottlenecked by the weakest stage, and the most common failure is silent: a correct chunk exists but sits outside the top-K recall window, so no downstream reranker or prompt can ever see it.

Chunking strategy is a retrieval-time decision masquerading as a preprocessing step. A chunk size that reads well may still produce vectors too generic to match a specific query, making retrieval tests the only valid tuning signal.

The tutorial's debugging order — data, chunks, recall, ranking, generation — is effectively a dependency graph. Skipping to prompt engineering when the chunker split a condition from its conclusion wastes time and masks the real fault.

Concepts & terms

Chunking

Splitting a long document into smaller text blocks so each block expresses a focused topic. The chunk becomes the unit for embedding, retrieval, and reranking; poor chunk boundaries degrade every downstream step.

Embedding

A model that converts text into a fixed-length vector of numbers representing its semantic meaning. The same embedding model must be used for both documents and queries so they occupy the same vector space.

Cosine Similarity

A measure of the angle between two vectors, ranging from -1 to 1. Values near 1 indicate high semantic similarity; it is the most common distance metric in vector retrieval.

Reranker (Cross-Encoder)

A model that takes a query and a candidate document together and outputs a relevance score. More accurate than embedding-based retrieval but computationally heavier, so it is applied only to a small set of recalled candidates.

BM25

A bag-of-words retrieval algorithm that ranks documents by term frequency and inverse document frequency. It complements vector search by handling exact keyword matches that embeddings may miss.

Agentic RAG

An extension of RAG where the system can query real-time APIs, choose among multiple data sources, or invoke tools based on intermediate retrieval results, rather than only answering from a static index.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗