What Actually Happens Inside LangChain's Document Loader and Text Splitter

Foreword

When I first started with RAG (Retrieval-Augmented Generation), what confused me most wasn't how the LLM generates answers, but what that whole chunk of "document processing" up front actually does.

What is a Document? How is it different from the PDF file on my computer? What does a Loader do? Why do we need to split? Why does chunkOverlap need to overlap? What exactly does the "Recursive" in RecursiveCharacterTextSplitter recurse over?

This article takes you through the RAG document preprocessing pipeline from scratch. I'll break down the source code line by line, using code + diagrams to explain every concept thoroughly.

1. Where Does the "Knowledge" in a Knowledge Base Come From?

A knowledge base stores knowledge, but knowledge sources are wildly diverse:

A Word document (.docx)
A PDF file (.pdf)
A Bilibili video (subtitles)
A URL (web page)
A reliable Twitter post

You can't just throw a PDF binary file at an LLM, nor can you directly vectorize a video.

So the first step: unify all formats into a single standard format.

2. File ≠ Document — The First Concept Many People Confuse

When I first started learning, I had this confusion too: isn't a file just a document?

A File is a physical-level thing — binary data stored on disk. A PDF is a binary stream, a Word document is actually a ZIP archive containing XML, and a video isn't even "text" at all.

A Document is a logical data structure defined by LangChain, with only two fields:

{
  pageContent: "This is the plain text content extracted from the file...",
  metadata: {
    source: "https://juejin.cn/post/xxx",
    title: "From Callback Hell to Elegant Async: The Evolution of Node.js File Operations"
  }
}

	File	Document
Level	Binary data on physical disk	Structured object in memory
Format	Wildly varied (.pdf .docx .mp4)	Unified: `{ pageContent, metadata }`
Usable directly?	❌ No, must be parsed first	✅ Recognized by all downstream steps
Analogy	Unopened delivery box	Items unpacked, sorted, and laid out

pageContent is the content itself, metadata is the "shipping label" describing the content. You can't handcraft a Document yourself — you must load the file through a Loader, which is responsible for extraction and encapsulation.

3. Loader: Turning Wildly Different Formats into a Unified Document

What a Loader Does

Raw files in various formats → Loader → Document (standard format)

A Loader does two things:

Selects the corresponding Loader based on file type — PDF uses PDFLoader, Word uses DocxLoader, web pages use CheerioWebBaseLoader
Outputs a standard Document — unified { pageContent, metadata }

Where Loaders Come From

Package Name	Maintainer	Description
`@langchain/core`	Official	Core abstractions: `BaseDocumentLoader` base class, `Document` class
`@langchain/community`	Community	Various concrete Loader implementations, anyone can contribute

4. Line-by-Line Breakdown of CheerioWebBaseLoader Source Code

CheerioWebBaseLoader is a web page loader provided by the LangChain community. Breaking down the name:

Part	Meaning
Cheerio	Uses the cheerio library under the hood to parse HTML
Web	Data source is a web page (URL)
Base	Base implementation class
Loader	Loader

It's a Class, Not a Function

Many people's first instinct is to think it's a function, but it's not:

// Creating an instance with the new keyword → this is a class
const cheerioLoader = new CheerioWebBaseLoader(
  'https://juejin.cn/post/7660707431753678854',
  { selector: '.main-area p' }
)

// .load() is a method on the instance
const documents = await cheerioLoader.load()

Why design it as a class? Because it has state to remember — the URL and selector passed during construction are stored on this.webPath and this.selector, used directly when .load() is called, without needing to be passed again.

Looking at the Source: Constructor

constructor(webPath, fields) {
    super();
    this.webPath = webPath;
    const { timeout, selector, textDecoder, headers, ...rest } = fields ?? {};
    this.timeout = timeout ?? 1e4;           // Default 10-second timeout
    this.caller = new AsyncCaller(rest);      // Handles retries and concurrency control
    this.selector = selector ?? "body";       // Default extracts entire body
    this.textDecoder = textDecoder;
    this.headers = headers;
}

Parameter	Default	What it does
`webPath`	Required	The URL of the web page to fetch
`timeout`	`10000` (10s)	HTTP request timeout duration
`selector`	`"body"`	CSS selector — only extracts text within matching elements
`headers`	`undefined`	Custom HTTP request headers

If you don't pass selector, the default is "body" — it grabs all text from the entire page, including nav bars, sidebars, and comment sections. Passing selector: '.main-area p' extracts only the article body paragraphs.

Core Method: One Line of .load(), Five Things Happen

Digging into the source code from node_modules:

async load() {
    // ① + ② Send HTTP request + parse into DOM tree
    const $ = await this.scrape();

    // ③ Extract page title (as metadata)
    const title = $("title").text();

    // ④ CSS selector extracts body text + ⑤ Wrap as standard Document
    return [new Document({
        pageContent: $(this.selector).text(),
        metadata: {
            source: this.webPath,
            title
        }
    })];
}

Note the return is an array [new Document(...)] — even though there's only one element. This is LangChain's unified design; all Loader .load() methods return a Document[] array.

Expanding _scrape(), inside is the actual network request and HTML parsing:

static async _scrape(url, caller, timeout, textDecoder, options) {
    const { load } = await CheerioWebBaseLoader.imports();

    // ① Send HTTP GET request (with timeout control)
    const response = await caller.call(fetch, url, {
        signal: timeout ? AbortSignal.timeout(timeout) : void 0,
        headers
    });

    // ② Parse HTML string into DOM tree, return $ query function
    return load(
        textDecoder?.decode(await response.arrayBuffer())
            ?? await response.text(),
        cheerioOptions
    );
}

Complete Data Flow

One line of .load() does five things:

① fetch(url)              → Send HTTP GET request (10s timeout)
② response.text()         → Get HTML string (193KB plain text)
③ cheerio.load(html)      → Parse into DOM tree in memory, return $ remote control
④ $(selector).text()      → CSS selector traverses tree, extracts plain text
⑤ new Document({...})     → Wrap as { pageContent, metadata }
              │
              ▼
      return [Document]    → Return as array

This is the meaning of encapsulation — hiding the tedious steps, giving you a clean entry point.

5. Manually Unpacking — Reproducing Step by Step with axios + cheerio

To thoroughly understand what the Loader does internally, I hand-wrote a crawl.mjs, reproducing it step by step at the lowest level:

What axios.get() Returns

const res = await axios.get(targetUrl)

Field	What it is	Actual Value
`res.data`	Response body	HTML string (193KB)
`res.status`	HTTP status code	`200`
`res.headers`	Response headers	`content-type: text/html; charset=utf-8`

Content-Type: text/html → axios automatically treats it as a string in res.data. If JSON was returned, it would be auto-parsed into a JS object.

Destructuring + Renaming

const { data: html } = await axios.get(targetUrl)
//      ↑     ↑
//   original  give it an alias: html

Extract res.data and rename it html — the variable name becomes more semantic, immediately clear it's an HTML string.

cheerio.load() — Turning a String into a DOM Tree

import * as cheerio from 'cheerio'

const $ = cheerio.load(html)

"<!doctype html><html lang="zh">..."  
              │    ← 193KB plain text, no structure
              ▼
    Build a DOM tree in memory:
    
               html
              /    \
           head    body
            │        │
          title    div (class="main-area")
            │      /        \
    "From Callback Hell..."  p          p
                    │          │
            "Foreword..."  "When I first started learning Node.js..."
              │
              ▼
    Return $ (the "remote control" for operating this tree)

Key insight: $ is not the DOM tree itself; it's a query function for operating the tree.

console.log(typeof $);  // 'function' ← It's a function!

One cheerio.load() builds one tree; you can build as many as you want — unlike a browser where there's only one global document.

CSS Selector Text Extraction

const pageContent = $('.main-area p').text()

Execution process:

$('.main-area p')
    ├── Find element with class="main-area"
    │   └── Find all <p> tags under it
    ├── Return node collection
    └── .text() iterates, strips tags, concatenates into plain text

.text() removes all HTML tags, directly concatenating multiple nodes.

What is "Frontend Thinking"?

Previously, scraping web pages with regex:

// Ugly and brittle, breaks as soon as HTML structure changes
const regex = /<p[^>]*class="[^"]*main-area[^"]*"[^>]*>([\s\S]*?)<\/p>/g

Regex treats HTML as a plain string to match — but HTML has nesting, arbitrary attribute order, whitespace anywhere; these are all regex disasters.

cheerio:

// Exactly like writing frontend jQuery
$('.main-area p').text()

	Regex	cheerio
Mindset	"Match strings starting with `<p` and ending with `</p>`"	"Select all p tags under class"
Operates on	Plain text string	DOM tree (hierarchical structure)
Reliability	Breaks when structure changes	Truly understands HTML syntax

Writing a scraper on the backend uses the exact same mental model as manipulating the DOM on the frontend. This is called "frontend thinking."

Four-Step Process Summary

1. HTML string runs in the command line, virtualizing a DOM object in memory
   → cheerio凭空 creates a DOM tree in Node.js memory, no browser needed

2. The DOM object is a tree structure, requesting allocation of relatively large memory space
   → 193KB HTML parsed into tens of thousands of node objects, each with parent-child pointers, more memory-intensive than the original text

3. CSS Selector searches within the tree
   → $('.main-area p') starts from the root node, traverses hierarchically, finds all matching nodes

4. cheerio lets JS developers complete scraping with a frontend mindset
   → No regex needed, write scrapers with jQuery feel

6. Splitter: Why Do We Need to Split?

The Document obtained by the Loader might be thousands or tens of thousands of words. Two problems:

Problem	Explanation
Too long	Model context window is limited
Poor precision	Searching for "callback hell" matches the whole article instead of precisely matching that paragraph

Split into Chunks. The core: control size while ensuring semantic integrity. Three questions: What to split by? How big? What if it cuts something off?

7. Deep Dive into RecursiveCharacterTextSplitter

What Exactly Does "Recursive" Recurse Over?

Separator degradation strategy:

Input text, target chunkSize: 400
              │
              ▼
    ① First try splitting by "。" (period) → complete sentences, best semantics
       │  Some sentence exceeds 400?
       ▼
    ② Then try splitting by "!" (exclamation mark)
       │  Still too big?
       ▼
    ③ Then try splitting by "?" (question mark)
       │  Still too big?
       ▼
    ④ Character-level hard split (fallback)

Degrades step by step from the most semantically complete boundary to the least semantic boundary. Periods yield complete sentences, exclamation and question marks do too. Only when there's no other choice does it hard split.

Code

const textSplitter = new RecursiveCharacterTextSplitter({
  chunkSize: 400,                // Each chunk max 400 characters
  separators: ["。", "!", "?"],   // Separator priority
  chunkOverlap: 100,             // Adjacent chunks overlap by 100 characters
})

const splitDocuments = await textSplitter.splitDocuments(documents)

Explaining the Three Parameters One by One

chunkSize (400): Maximum character count, not an exact value. If a sentence is 410 characters but a complete sentence, it will be kept. Semantics first, size second.

separators (["。", "!", "?"]): Priority list, order matters. Chinese uses periods, exclamation marks, question marks (natural sentence boundaries); English defaults to ["\n\n", "\n", " ", ""] (paragraph → line → word → character).

chunkOverlap (100): The most important yet most easily overlooked parameter.

Why Overlap? An Example

No overlap:

chunk1: "Callback hell is a headache. To solve it, there appeared"
chunk2: "Promise. Promise makes async code more elegant."

Search "what appeared" → chunk1 matches, but it doesn't contain "Promise"! The answer is at the head of chunk2, exactly cut off, lost ❌

Overlap (100 characters):

chunk1: "Callback hell is a headache. To solve it, there appeared Promise."
chunk2: "To solve it, there appeared Promise. Promise makes async code more elegant."
                       ↑   Overlap region

Search "what appeared" → both sides have "appeared Promise", both can answer ✅

Overlap = leaving a fallback for key context at the cut boundary. The content near that cut is copied to both sides; no matter which side is hit, information isn't lost.

Overlap Isn't "More is Better"

"For semantic integrity, use less chunkOverlap" — this "less" is key.

Consequences of too large (300): One-third of each segment is duplicated, wasting storage, wasting Embedding tokens, massive redundancy in retrieval results.

Rule of thumb: 10%~25% of chunkSize. For 400 → overlap 50~100 is reasonable.

8. Comparing Two Files — Same Task, Two Abstraction Levels

	index.mjs (Taking the bus)	crawl.mjs (Driving)
Method	Loader encapsulated	axios + cheerio manually
Send request	Loader internal fetch	Manually `axios.get()`
Parse HTML	Loader internal cheerio	Manually `cheerio.load()`
Extract content	Pass `selector` config	Manually `$(selector).text()`
Return	Standard Document	Plain string
Downstream usable	✅ Directly feed to Splitter	❌ Must manually wrap metadata

The point of crawl.mjs isn't "being easy to use" — it's letting you see clearly what the Loader does internally.

9. Complete Data Flow

Various knowledge files (PDF / Word / URL / Video / Twitter)
              │
              ▼
          Loader (Loading)
      Select corresponding Loader, call .load()
              │
              ▼
      Document (Standard Format)
      { pageContent, metadata }
              │
              ▼
          Splitter (Splitting)
      Recursively degrade separators to ensure semantics
      overlap preserves context at cut boundaries
      chunkSize controls each chunk's size
              │
              ▼
      Chunks (Small Document[])
      Semantically complete, size-controlled, adjacent overlap
              │
              ▼
      Embedding (Vectorization)
              │
              ▼
      Vector Database (Storage)
              │
              ▼
      Retrieval (Retrieval)
      User query → Vectorize → Similarity search → Recall Chunk → LLM Generation

10. ESM Module System Quick Reference

Syntax	Meaning	Analogy
`export default`	Default export, only one	The most-used hammer in the toolbox
`export`	Named export, can be many	Other tools
`import xxx from`	Import default, name arbitrary	Reach for the hammer
`import { xxx }`	Import named, name must match	Grab the screwdriver by name
`import * as xxx`	Import all, bundle	Take the entire toolbox
`const { a: b } = obj`	Destructure + rename	Take `obj.a` and call it `b`

Afterword

Once you understand Loader and Splitter, you'll find the design truly exquisite:

Loader — Unifies wildly different sources into { pageContent, metadata }. This is the cornerstone that makes the entire RAG pipeline "universal."
Splitter — Recursive separator degradation + overlap to preserve semantics. Finds the optimal balance between "semantic integrity" and "size controllability," not just a simple hard split by character count.

Finally, a quote from the readme:

No longer coding, hand it over to AI. The core capabilities of Vibe coding: asking good questions (Prompt), providing accurate context (Context), harnessing and deploying Agent products (Harness + FDE), designing stably running Loops. Rapidly grow into an AI Architect.

Understanding the underlying principles allows you to better harness the upper-level encapsulations.