Agent · AI Programming · OpenAI

RAG Isn't Just Retrieval: Chunking, Re-Ranking, and Graph Strategies That Make Agents Actually Work

By 怕浪猫 · Jul 24, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Most RAG demos stop at a vector database and call it done. Real deployments break on multi-hop questions, stale documents, and retrieval that returns semantically similar but useless text. The techniques here—hybrid search, re-ranking, parent-child chunking, and knowledge graphs—are what separate a toy from a system that answers questions correctly in production.

Summary

Pure vector search fails when semantic similarity doesn't equal answer relevance. Hybrid retrieval combines dense embeddings with BM25 keyword matching, then a re-ranking model like Cohere's rerank-v3.5 filters the top candidates. Query rewriting and HyDE—generating a hypothetical answer to use as the search query—further improve recall.

Knowledge graphs fill a gap that vectors can't touch: multi-hop relational queries. GraphCypherQAChain translates natural language into Cypher queries against Neo4j, and Microsoft's GraphRAG extracts entity communities, summarizes them, and retrieves within relevant clusters. For documents, parent-child chunking retrieves on small chunks for precision but returns larger parent chunks for context.

Production RAG also needs to solve "lost in the middle" attention drop-off, multi-hop reasoning through iterative retrieval loops, and conflicting information across sources. Incremental indexing with content hashing and versioned rollback keeps the knowledge base current without full rebuilds—an outdated knowledge base is worse than none.

Takeaways

— Semantic similarity does not equal answer relevance; pure vector retrieval often returns topically close but factually wrong documents.

— Hybrid search combines dense vector retrieval with BM25 keyword matching, weighted at 50/50, to catch exact term matches that embeddings miss.

— Re-ranking with a model like Cohere rerank-v3.5 takes the top-K candidates from retrieval and re-sorts them by actual relevance to the query.

— Parent-child chunking retrieves on small 200-character chunks for precision but returns the surrounding 1000-character parent chunk for context.

— HyDE generates a hypothetical answer to a question and uses that answer as the retrieval query, often surfacing more relevant documents than the raw question.

— Knowledge graphs handle multi-hop relational queries—like "who co-founded PayPal with Musk and what else did they start"—that vector search cannot.

— LLMs pay the least attention to information in the middle of a context window; reordering documents by alternating relevance prevents critical chunks from landing there.

— Multi-hop reasoning requires iterative retrieval: retrieve, assess if the question is answerable, and generate a follow-up sub-query if not.

— An outdated knowledge base is more dangerous than none; incremental indexing with MD5 content hashing detects changes and updates only what shifted.

— Versioned knowledge bases with rollback let you revert to a known-good document state when an update introduces errors.

Conclusions

RAG's hardest problems are not about embeddings or vector databases—they are about information architecture: how you chunk, how you order results, and how you handle conflicting sources.

The "lost in the middle" phenomenon means that even perfect retrieval can fail if the most relevant chunk lands in the LLM's attention dead zone, making document ordering a surprisingly high-leverage fix.

HyDE inverts the retrieval paradigm: instead of hoping the query matches the right document, you generate a plausible answer and search for documents that resemble it, which often works better for open-ended questions.

Knowledge graphs and vector search are not competitors—they solve different query shapes. A production RAG system needs both, plus a router that decides which path a question takes.

Most RAG tutorials treat the knowledge base as static, but real-world documents change constantly. The engineering effort shifts from building the index to maintaining it over time.

Concepts & terms

RAG (Retrieval-Augmented Generation)

A technique that gives an LLM access to an external knowledge base. At query time, relevant documents are retrieved and inserted into the prompt so the model can ground its answer in provided facts rather than relying solely on training data.

Parent-Child Chunking (Small-to-Big Retrieval)

A chunking strategy where documents are split into small child chunks for precise similarity search, but the system returns the larger parent chunk that contains the child, preserving surrounding context.

BM25

A bag-of-words retrieval algorithm that ranks documents by term frequency and inverse document frequency. It excels at exact keyword matching, complementing the semantic matching of vector embeddings.

Re-ranking

A second-pass scoring step where a more expensive, cross-encoder model evaluates the relevance of each retrieved document to the query and re-sorts the list, keeping only the top-N most relevant.

HyDE (Hypothetical Document Embeddings)

A technique where an LLM generates a hypothetical answer to a query, and that synthetic answer is embedded and used as the retrieval query. It often surfaces more relevant documents than embedding the raw question.

GraphRAG

Microsoft's approach that extracts entities and relationships from documents, builds a community graph, generates summaries per community, and retrieves by first locating relevant communities before searching within them.

Lost in the Middle

A documented LLM behavior where information placed in the middle of a long context window receives the least attention, making document ordering critical when multiple chunks are fed into the prompt.

Multi-Hop Reasoning

A question that requires chaining multiple facts together—e.g., 'birthplace of the wife of the 46th president'—demanding iterative retrieval where each hop's answer informs the next query.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗