The Recursive Splitter That Respects Your Sentences

Knowledge Base Preprocessing in Practice: Full-Chain Analysis from URL Loading to Semantic Chunking

This article does not pile up APIs but starts from underlying principles to dissect the design philosophy of document loading and splitting within the LangChain ecosystem, complete with runnable code.

1. The "Raw Material" Dilemma of Knowledge Bases

When building a RAG (Retrieval-Augmented Generation) application, the first problem we face is not model selection, but how to "feed" knowledge into the vector database. Knowledge sources are wildly diverse:

A .docx office document
A .pdf academic paper
Subtitles from a Bilibili video (or even audio transcription)
A Twitter thread
A dynamically rendered web page URL

This heterogeneous data must be unified into a certain structured document object before vectorization. LangChain provides a standard abstraction — Document, which contains two core fields:

pageContent: plain text content (the input for vectorization)
metadata: contextual information like source, time, and author

But generating a Document directly from a raw file is not trivial, so the Loader was born.

2. Loader: From "Diverse Forms" to "Unified Standard"

The Loader's responsibilities are clear:

Adaptation: Choose the appropriate parsing strategy based on the file/URL/byte stream.
Standardization: Output a Document[] array for subsequent processing.

The LangChain community provides 180+ Loaders (in the @langchain/community package), covering common formats. The community-driven design means we can even implement and contribute our own Loaders.

The official core package @langchain/core maintains the base interfaces, while @langchain/community hosts most concrete implementations. This layering reduces core library bloat and encourages ecosystem prosperity.

3. The Underlying Logic of Crawler-type Loaders: Cheerio as an Example

When we face a URL, we usually need to crawl the web page content. But a web page is an HTML structure; we only need the main text, and ideally, we can target a specific area (like the article paragraphs). CheerioWebBaseLoader is designed for this.

A Three-Step Working Principle

Send an HTTP request: Fetch the complete HTML string of the target URL.
Parse the DOM: Use the cheerio library (similar to server-side jQuery) to load the HTML and form a queryable DOM tree.
Extract with CSS selectors: Select elements using the selector we specify (e.g., '.main-area p'), call .text() to get the plain text content, and assemble it into a Document.

The key here is cheerio's "frontend mindset" — it allows backend programmers to precisely locate content using CSS selectors without writing manual regex or complex DOM traversal. Compared to regex parsing, cheerio is more stable and maintainable because selectors align with frontend development habits and can handle most page structure changes.

Code Snippet

import { CheerioWebBaseLoader } from "@langchain/community/document_loaders/web/cheerio";

const loader = new CheerioWebBaseLoader(
  'https://juejin.cn/post/7660707431753678854',
  { selector: '.main-area p' }  // Only grab the article's main body paragraphs
);

const docs = await loader.load();
// docs is a Document array, each element corresponds to the content of a <p> tag

Note: selector is optional; if not provided, it extracts text from the entire <body>. But doing so introduces a lot of noise (navigation, sidebars, etc.), so precise selection is key in engineering practice.

4. Splitting: Why Can't We Just Treat the Whole Web Page as One Document?

Even if we successfully load a web page, its text length often far exceeds the model's context window (e.g., 4k, 8k). Moreover, during vector retrieval, we want the smallest retrieved unit to have independent semantics — a paragraph, a viewpoint, an argument unit, not the entire article.

Therefore, chunking is a necessary step before vectorization. But chunking is not simply truncating by character count, because semantic integrity is the primary goal.

4.1 Choosing Semantic Boundaries

In natural language, the basic semantic boundary is usually the sentence. Sentences end with periods, question marks, and exclamation marks. Commas and semicolons only indicate brief pauses and are not suitable as breakpoints, as they would sever causal or progressive relationships.

Therefore, a good splitter will prioritize these strong separators.

4.2 The Recursive Chunking Algorithm (RecursiveCharacterTextSplitter)

The RecursiveCharacterTextSplitter provided by LangChain adopts a recursive probing strategy. Its core logic is as follows:

Accept a list of separators sorted by priority (e.g., ['。', '！', '？', '\n\n', '\n', ' ', '']).
First, try to split using the first separator (usually a period). If the length of each resulting chunk is less than chunkSize, stop.
If a chunk is still too long, recursively use the next separator on that chunk to continue splitting.
The final result ensures each chunk is as close to chunkSize as possible, while preferentially breaking at semantic boundaries.

This recursive method is much smarter than simple fixed-length character truncation because it respects the hierarchical structure of natural language (sentence → paragraph → chapter).

4.3 The Significance of Overlap

Even with the best separators, "semantic regret" is unavoidable — for example, a long sentence exceeding chunkSize is forcibly truncated mid-sentence, causing the first chunk to lose the second half and the second chunk to lose the first half, even though the two parts are semantically closely linked.

The chunkOverlap parameter allows adjacent chunks to retain a portion of overlapping content. It is usually set to 10%~25% of chunkSize. This way, even if a truncation point occurs, contextual information can be transmitted through the overlap area. During retrieval, whether the preceding or following chunk is hit, the complete logical context can be obtained.

Underlying implementation: Overlap is achieved by copying a specified length of characters from the end of the previous chunk to the beginning of the next, ensuring no boundary information is lost.

5. Complete Code Line-by-Line Analysis

Below is the index.mjs file we actually ran, with comments for an in-depth analysis of every detail.

import "dotenv/config";          // Load environment variables (like OPENAI_API_KEY), reserved for subsequent vectorization
import "cheerio";               // Ensure cheerio is installed; CheerioWebBaseLoader depends on it internally

// Import Web Loader from @langchain/community
import {
  CheerioWebBaseLoader
} from "@langchain/community/document_loaders/web/cheerio";

// Import recursive splitter from @langchain/textsplitters
import {
  RecursiveCharacterTextSplitter
} from '@langchain/textsplitters';

// --- Step 1: Instantiate Loader, specifying URL and CSS selector ---
const cheerioLoader = new CheerioWebBaseLoader(
  'https://juejin.cn/post/7660707431753678854', // Target article
  {
    selector: '.main-area p',   // Juejin article body paragraphs are usually under this class
  }
);

// --- Step 2: Load to get a Document array (here, each <p> becomes a Document) ---
const documents = await cheerioLoader.load();
// At this point, the length of documents depends on the number of paragraphs in the article.
// Each element's pageContent is the paragraph text, and metadata contains the URL, etc.

// --- Step 3: Instantiate the splitter, configure chunking parameters ---
const textSplitter = new RecursiveCharacterTextSplitter({
  chunkSize: 400,              // Target size for each chunk (in characters)
  separators: ['。', '！', '？'], // Force Chinese sentence separators, priority from high to low
  chunkOverlap: 100,           // Overlap character count to prevent semantic truncation
});

// --- Step 4: Execute splitting on Documents, returning a smaller Document array ---
const splitDocuments = await textSplitter.splitDocuments(documents);

// Output the result for subsequent vectorization and storage
console.log(splitDocuments);

Detailed Considerations

Why are only three punctuation marks set for separators? Because Chinese articles use periods, question marks, and exclamation marks as the most natural sentence boundaries. If chunks still exceed chunkSize after splitting with these three separators, RecursiveCharacterTextSplitter will automatically fall back to default newline characters, spaces, etc. (there is an internal default fallback list if not specified in the constructor). We could explicitly pass a more complete separator hierarchy, but here, for demonstration purposes, semantic priority is emphasized with just three.
Is chunkOverlap=100 reasonable? For a 400-character chunk, a 25% overlap rate can effectively cover transitions between sentences. However, note that overlap increases storage volume and may produce redundant matches during retrieval, requiring a trade-off between precision and recall.

6. Flowchart: The Data Pipeline from URL to Pre-Vectorization

graph TD
    A[Original URL] -->|HTTP Request| B(HTML String)
    B -->|cheerio.load + selector| C[Extract Paragraph Text]
    C -->|Generate one Document per paragraph| D[Document Array]
    D -->|Input to RecursiveCharacterTextSplitter| E{Try splitting with highest-priority separator}
    E -->|Chunk size <= chunkSize?| F[Accept the chunk]
    E -->|Chunk too large| G[Recursively use next-level separator]
    G --> F
    F -->|Add chunkOverlap| H[Final Chunked Documents]
    H -->|Output| I[Vectorization & Storage]

7. The Value Migration of Programmers in the AI Era

After reading the above technical details, let's step back from the code and consider a more macro-level question: In an era where AI can write code automatically, where does a programmer's true value lie?

In the Vibe Coding era, the ability to ask questions (Prompt Engineering) and build context (Context Provision) becomes more important than memorizing syntax.
Mastering Agents: We need to design stable-running loops, handling exceptions, retries, and state management, rather than personally writing every line of parsing logic.
Architectural Thinking: Like the splitting strategy in this article, decisions on which separators to choose, what overlap rate to set, and how to tune them directly impact the retrieval quality of a RAG system — this is the responsibility of an AI Architect, not an ordinary CRUD engineer.

Splitting seems simple but actually embodies a deep understanding of language structure and information retrieval. When we hand this task to an AI for generation, we must be able to review, tune, and deploy the entire process. This is the core competency of a new-era engineer.

8. Summary and Outlook

Loader is the "adapter" for a knowledge base, shielding format differences and outputting a unified Document.
CheerioWebBaseLoader uses CSS selectors for precise web content extraction, an efficient solution for crawling scenarios.
Splitting is not simple truncation but balances length constraints with semantic integrity through recursive separators and overlapping windows.
Although the code is short, there are engineering trade-offs behind every parameter, worthy of repeated experimentation (e.g., adjusting chunkSize and overlap to suit different domain texts).

Subsequently, we can pass splitDocuments to an Embeddings model (like OpenAIEmbeddings) and store them in a vector database (like Pinecone, Chroma), ultimately building a complete RAG chain. This article focuses only on the preprocessing stage, but it is the cornerstone of the entire chain — garbage in, garbage out; high-quality Document chunking is the beginning of high-quality retrieval.