跪拜 Guibai
← Back to the summary

The RAG Ingestion Pipeline Is Where Your Retrieval Quality Is Won or Lost

RAG's Knowledge Refinery: The Engineering Path from Raw Web Pages to Retrievable Knowledge Chunks

If we compare RAG (Retrieval-Augmented Generation) to a knowledge factory, then the article from v040 (which you can review here) discussed "how the system accurately finds materials and generates answers after a user asks a question," which belongs to the factory's "sales and distribution" segment. Today, we turn our lens to a more upstream, more hidden, yet life-or-death segment for the factory—"raw material intake."

Before knowledge enters the vector database "warehouse," it must go through a complex "refining" process: Raw Material → Extraction → Purification → Splitting → Packaging → Warehousing

The quality of this upstream chain directly locks in the performance ceiling of the RAG system. The industry often says "Garbage in, garbage out," but in the RAG field, it's even more brutal: Incorrect knowledge formatting renders even the most powerful retrieval and generation models helpless.

This article will walk you through the core path of this knowledge refining, delving into document standardization, the pitfalls of web page extraction, the wisdom of recursive splitters, and how to forge a teaching demo into a production-ready pipeline.

1. Core Insight: RAG's Upstream Determines the Downstream Ceiling

Common RAG diagrams often start drawing from Embedding onwards, as if text is inherently clean and structured. But the "raw materials" in reality are chaotic: they might be ad-filled web pages, poorly formatted PDFs, or unstructured conversation logs.

A complete data ingestion chain looks like this:

graph TD
    A[Raw Material<br>Web/PDF/DB] --> B(Loader Extraction);
    B --> C{Quality Check};
    C -- Passed --> D[Cleaning & Standardization<br>Noise Removal];
    C -- Failed --> E[Log Failure/Alert];
    D --> F[Splitter<br>Semantic Unit Segmentation];
    F --> G[Package as Knowledge Chunk<br>Chunk + Metadata];
    G --> H[Vectorization & Indexing];
    
    style A fill:#f9f,stroke:#333,stroke-width:2px
    style F fill:#ccf,stroke:#333,stroke-width:2px
    style G fill:#9f9,stroke:#333,stroke-width:2px

Every link can become a "quality funnel," causing information distortion:

Link Common Mistake Fatal Impact on Downstream
Loader Extraction Choosing the wrong tool (e.g., using a static loader for dynamic web pages) Cannot get the main text at all; you can't make bricks without straw
Selector Selection too broad (grabbing navigation, ads) or too narrow (missing titles, lists) Vector semantics drowned by noise, or key information missing
Cleaning Not removing duplicate footers, disclaimers Retrieval results heavily duplicated, wasting context window
Splitting (Splitter) Chunks too large, mixing multiple topics Retrieval recall is coarse, unable to hit precisely
Chunks too small, breaking causality and references Model receives semantic fragments, unable to reason
No overlap set Key information falls exactly on the boundary, cut in half

Therefore, a simple but crucial conclusion is: Problems during reading (Retrieval & Generation) might still be remedied with better Prompts or re-ranking models, but errors during writing (Ingestion), once stored, become permanent, hard-to-trace "data pollution."

2. From Chaos to Order: Document — The Unified Standard of the Knowledge Refinery

No matter the source of knowledge, before entering the core processing pipeline, it must be transformed into a unified data structure. In the world of LangChain, this standard is the Document.

It is extremely concise, with only two core fields:

  1. pageContent: The pure text content that actually participates in splitting, vectorization, and retrieval.
  2. metadata: Structured information describing the content, typically not vectorized, but present throughout the entire lifecycle.
// A standard Document object
new Document({
  pageContent: 'Refund applications must be submitted within seven days of order completion...',
  metadata: {
    source: 'help/refund-policy.pdf', // Source
    page: 12,                         // Location
    section: 'Refund Rules',          // Section
    updatedAt: '2026-07-01'           // Timeliness
  }
})

Why can't you just store an entire PDF file as one Document? Imagine a 100-page product manual compressed into a single vector. When a user asks "How do I get a refund?", the semantics represented by this vector are the "average semantics of the entire manual." It cannot guide the user to the "Refund Rules" on page 12. Therefore, a Document is the standard input, but far from the final indexing unit.

3. Extraction (Loader): HTTP 200 Doesn't Mean You Got the Content

The code example uses CheerioWebBaseLoader with the CSS selector .main-area p to scrape a web page. These few lines of code are precisely where countless RAG projects begin to crash in production environments.

await loader.load() executing successfully only means the request didn't crash, absolutely not that content extraction was successful. What you get might be:

3.1 What Axios + Cheerio Can and Cannot Do

3.2 Production-Grade Extraction Strategy: Validation, Fallback, and Compliance

In a production environment, validation must immediately follow extraction:

// 1. Content validity check
if (html.includes('waf-jschallenge') || html.includes('Please wait...')) {
  throw new Error('Request blocked, returned a challenge page');
}

const nodes = $('.main-area p');
if (nodes.length === 0 || nodes.text().length < 200) {
  throw new Error('Main text extraction failed: Selector invalid or content empty');
}

// 2. Strategy Selection
// Static page + authorized -> Cheerio (High performance)
// Dynamic rendering + authorized -> Playwright/Puppeteer (Simulate browser)
// Large-scale scraping -> Prioritize official APIs, strictly comply with robots.txt and ToS

Remember, technical solutions can never be a reason to bypass compliance and copyright.

4. Splitting (Splitter): The Core Craft of Knowledge Refining

The extracted "crude oil" (a large Document) needs to be split into standardized "barrels of oil" (multiple small Chunks) to be efficiently processed by the vector retrieval "fractionating tower."

4.1 Goal: The Smallest Complete Semantic Unit

The ideal state for each Chunk is: Contains one and only one core topic, has complete context, and can independently answer questions related to that topic. For example, an article about "Product Features" should be split into independent Chunks like "Installation," "Configuration," "API Authentication," and "Troubleshooting."

4.2 RecursiveCharacterTextSplitter: An Elegant Recursive Tactic

This is the most commonly used splitter. The word "recursive" behind it represents a greedy and elegant fallback strategy.

Its workflow can be understood as follows:

  1. Greedy Attempt: First, try to split the text using the highest priority separator you specified (e.g., paragraph \n\n), hoping to get chunks that fit the size (chunkSize).
  2. Recursive Fallback: If a paragraph is still too long, it will "recursively" process this "problem paragraph," trying to cut it again using the next level separator (e.g., period .).
  3. Final Fallback: If even the last separator (usually an empty string '') fails to control the size, it will be forced to hard-cut by character count to prevent the program from crashing.

Therefore, the order of the separators array is a reflection of your understanding and preference for text structure. For Chinese articles, a recommended configuration is:

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 400,
  chunkOverlap: 80,
  separators: ['\n\n', '\n', '。', '！', '？', '；', '，', ''],
  // The final '' is crucial, acting as a safety valve against excessively long continuous characters
});

4.3 chunkOverlap: "Context Insurance" for Boundary Information, Not a Silver Bullet

Splitting is like cutting a cake; there's always some "cream" that sticks to the knife. The role of chunkOverlap is to let two adjacent Chunks share a portion of this "cream" (context), ensuring that key information like references, causality, and definitions are not cut off exactly at the boundary.

But it cannot solve fundamental problems, such as the extracted content itself being wrong, or a Chunk mixing multiple unrelated topics. The correct order of use is: First optimize extraction quality → Then design separator priority → Finally use Overlap as a supplementary buffer.

5. From Demo to Production: An Engineering Practice Checklist for the Refinery

Teaching examples often stop abruptly after splitDocuments, but the production battle has just begun. Below are the core capabilities a production-ready document processing pipeline should possess:

5.1 Quality Gates and Observability

Never trust any unverified data. Before writing to the vector store, a "quality gate" must be set:

const validDocs = documents.filter(doc => doc.pageContent.trim().length >= 200);
if (validDocs.length === 0) { /* Trigger alert, terminate process */ }

// Log key metrics for monitoring and debugging
const metrics = {
  source_url: url,
  extracted_text_length: doc.pageContent.length,
  chunk_count: chunks.length,
  // ...
};

5.2 Idempotency and Incremental Updates

You cannot delete and rebuild all documents every time. Calculate a content hash for each Document and Chunk. If the hash hasn't changed, skip the update; if it has, delete the old Chunk and write the new one. This significantly saves costs and enables smooth incremental updates.

5.3 Permissions and Security (Design First!)

Permissions are not an afterthought for the generation layer; they should be bound at ingestion time. Once a chunk is retrieved, it can potentially be leaked. Writing permission information (like tenantId, role) into metadata and enforcing it as a filter condition during retrieval is the only way to build a secure enterprise-grade RAG system.

6. How to Evaluate Splitting Quality: Not Just "It Split," But "It Split Correctly"

Just because splitter.splitDocuments() returned an array doesn't mean the splitting job is qualified. Evaluation should cover at least three dimensions:

  1. Structural Dimension: Are titles and body text separated? Are code blocks and tables truncated? What is the proportion of empty or overly long chunks?
  2. Retrieval Dimension: Prepare a set of real questions and check if the Chunk containing the correct answer consistently appears in the Top-K retrieval results. This is the most core metric.
  3. Generation Dimension: Is the final answer "evidence-based"? Are the cited sources correct? Is the context wasting a large number of Tokens due to duplicated Overlap?

The optimal splitting parameters are not calculated by a formula but are weighed and balanced through repeated validation on your business dataset across these three dimensions.

Conclusion: RAG's Success Begins with Respect for the Upstream

The code practice on the thirty-seventh day seems to be just configuring a Loader and a Splitter, but it contains the most fundamental philosophy of the RAG domain:

v040 explored "how to efficiently find the right gold ingot," while this piece is dedicated to elaborating on "how to produce high-quality gold ingots." Only when the upstream refinery can stably output high-purity, high-spec knowledge chunks can the downstream retrieval and generation engines truly exert their power. The upper limit of RAG is destined from the very beginning by this hidden yet critical refining chain.

High-Frequency Interview Points Quick Reference