A Two-Step RAG Walkthrough: From a Handwritten Story to Crawling Juejin Articles
Implementing RAG from Scratch: Crawl Juejin Articles and Let LLMs Help You Understand Any Webpage
What is RAG?
RAG (Retrieval-Augmented Generation) is the core solution to the "hallucination" problem in LLMs.
Simply put: an LLM's training data is limited. When you ask it a question outside its training set (like an article published just yesterday), it will "confidently spout nonsense" — this is hallucination.
The idea behind RAG is straightforward: first retrieve relevant knowledge, then let the LLM answer based on that knowledge.
User Question → Vector Retrieval → Find Relevant Document Chunks → Insert into Prompt → LLM Generates Answer
This article walks you through the complete pipeline with two progressive hands-on cases, from a manual knowledge base to a fully automated web-crawling RAG.
Case 1: Manual Knowledge Base RAG
We start with the simplest scenario — manually writing a friendship story about "Guangguang and Dongdong" and letting the LLM answer questions based on this story.
Core Code Structure
import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai";
import { Document } from "@langchain/core/documents";
import { MemoryVectorStore } from "@langchain/classic/vectorstores/memory";
// 1. Manually construct a knowledge base (7 Documents)
const documents = [
new Document({
pageContent: `Guangguang is a lively and cheerful little boy...`,
metadata: { chapter: 1, character: "Guangguang", type: "character intro", mood: "lively" },
}),
// ... 7 story segments in total
];
// 2. Vectorize and store in memory
const vectorStore = await MemoryVectorStore.fromDocuments(documents, embeddings);
// 3. Build a retriever
const retriever = vectorStore.asRetriever({ k: 3 });
// 4. Ask → Retrieve → Augment Prompt → Generate Answer
const retrievedDocs = await retriever.invoke("What are Guangguang and Dongdong each good at?");
const context = retrievedDocs.map((doc, i) => `[Segment ${i+1}]\n${doc.pageContent}`).join("\n\n----\n\n");
const prompt = `Answer the question based on the following story segments:\n${context}\n\nQuestion: ${question}\nAnswer:`;
const response = await model.invoke(prompt);
Document: The "Standard Carrier" of Knowledge
Document is the core data structure in LangChain. Each Document represents a segment (chunk) in the knowledge base:
| Field | Purpose |
|---|---|
pageContent |
Stores the actual text, which will be embedded into vectors for semantic search |
metadata |
Attached labels (chapter, character, type, etc.), not involved in vector calculation, usable for filtering retrieval |
In the full RAG pipeline, Document is the unified data format throughout:
Raw Text Slicing → Document[] → Embedding Vectorization → Store in Vector DB → Retrieval Returns Document[] → Extract Content into Prompt
Case 2: Automated Web Crawler RAG (loader-and-splitter.mjs)
Manually writing a knowledge base is unrealistic. Real RAG requires automatically loading documents from various sources, auto-slicing, and auto-vectorizing.
This case implements: crawl a Juejin article → auto-slice → vectorize → LLM Q&A.
Complete Flow Diagram
Juejin Article URL
→ CheerioWebBaseLoader (crawl + parse)
→ RecursiveCharacterTextSplitter (smart slicing)
→ OpenAIEmbeddings (vectorization)
→ MemoryVectorStore (store in in-memory vector DB)
→ retriever (retrieval)
→ LLM (generate answer)
Detailed Explanation of Three Core APIs
1. CheerioWebBaseLoader — Web Page Loader
import { CheerioWebBaseLoader } from "@langchain/community/document_loaders/web/cheerio";
const loader = new CheerioWebBaseLoader(
"https://juejin.cn/post/7233327509919547452",
{ selector: '.main-area p' } // CSS selector, only extract body paragraphs
);
const documents = await loader.load();
Underlying Principle: Uses cheerio (server-side jQuery) to parse HTML. You can use CSS selectors to precisely target the content you want to extract, just like manipulating the frontend DOM.
Key Configuration: selector determines which DOM nodes to scrape. Here .main-area p only takes <p> tags from the article body area, automatically filtering out noise like navigation bars and comment sections.
2. RecursiveCharacterTextSplitter — Recursive Text Splitter
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
const splitter = new RecursiveCharacterTextSplitter({
chunkSize: 400, // Max characters per chunk
chunkOverlap: 50, // Overlapping characters between adjacent chunks
separators: ['。', '，', '！', '？'] // Separator priority
});
const chunks = await splitter.splitDocuments(documents);
The role of the three parameters:
chunkSize: The maximum number of characters per text block. This is an upper limit, not a fixed value.chunkOverlap: The number of overlapping characters between adjacent blocks. For example, the last 50 characters of chunk1 and the first 50 characters of chunk2 are the same, preventing key information from being "cut off" at the boundary.separators: Separators sorted by priority. It prefers to cut at periods, then commas, then exclamation marks and question marks, ensuring cuts happen at natural semantic boundaries as much as possible.
Do separators and chunkSize conflict?
No. RecursiveCharacterTextSplitter uses a "cut first, then merge" strategy:
- First, use
。to cut all sentences apart - Greedy merge: Sentence A + Sentence A + Sentence A → if it doesn't exceed 400 characters, keep adding
- When adding a sentence would exceed 400, stop, this chunk is complete
- Start a new chunk from the next sentence
So ultimately a chunk may contain 2-4 complete sentences, with a size close to but not exceeding 400 characters.
3. MemoryVectorStore — In-Memory Vector Database
import { MemoryVectorStore } from "@langchain/classic/vectorstores/memory";
const vectorStore = await MemoryVectorStore.fromDocuments(splitDocuments, embeddings);
const retriever = vectorStore.asRetriever({ k: 2 });
MemoryVectorStore stores all vector data in memory, suitable for learning and prototype validation.
Two core methods:
| Method | Purpose |
|---|---|
fromDocuments(docs, embeddings) |
Embed a list of documents and store them in memory |
asRetriever({ k: 2 }) |
Wrap the vector store as a retriever, k=number of results to return each time |
What actually happens during retrieval:
- User question → embedding → question vector
- Calculate cosine similarity between the question vector and all document vectors in the store
- Cosine values closer to 1 indicate higher semantic similarity, closer to 0 indicates irrelevance
- Sort by similarity, return the Top K Documents
Comparison of Common LangChain Splitter Behaviors
Different Splitters have vastly different cutting strategies; choosing the right tool is important:
| Splitter | Strategy | Characteristics |
|---|---|---|
RecursiveCharacterTextSplitter |
Recursively cut by separator priority, greedily merge to approach chunkSize | Maintains sentence integrity, most commonly used |
CharacterTextSplitter |
Cut by fixed character count + sliding window | Simple and crude, may truncate sentences |
TokenTextSplitter |
Cut by token count | Precisely controls token consumption, but doesn't care about semantic boundaries |
MarkdownHeaderTextSplitter |
Cut by Markdown heading levels | Suitable for structured documents |
HTMLHeaderTextSplitter |
Cut by HTML heading tags | Suitable for web documents |
Key Difference: Only RecursiveCharacterTextSplitter "first cuts all by separators, then greedily merges to fill chunkSize"; others cut chunk by chunk with a fixed window. Therefore, Recursive's chunk quality is usually the highest.
Building the Prompt and Calling the LLM
After retrieving relevant documents, insert them into the Prompt:
const context = retrievedDocs
.map((doc, i) => `[Segment ${i+1}]\n${doc.pageContent}`)
.join("\n\n----\n\n");
const prompt = `You are an article reading assistant, answer based on the article content:
Article Content:
${context}
Question:
${question}
Answer:`;
const response = await model.invoke(prompt);
This is the core of Augmented in RAG: the Prompt given to the LLM is not just the user's question, but also includes the retrieved relevant context.
Regarding Compatibility with Domestic Models
The code uses @langchain/openai, but the actual calls might be to domestic models like Tongyi Qianwen (Qwen) or DeepSeek. The key lies in:
const model = new ChatOpenAI({
apiKey: process.env.OPENAI_API_KEY, // API Key for the domestic model
configuration: {
baseURL: process.env.OPENAI_BASE_URL // API address for the domestic model
},
});
Major domestic LLMs (Alibaba Qwen, DeepSeek, Zhipu GLM, Moonshot Kimi, 01.AI Yi, etc.) all provide OpenAI-compatible interfaces. You only need to change baseURL and apiKey; the code remains unchanged.
Summary
From the two cases, we can see the complete RAG pipeline:
- Loader: Load documents from various sources (web pages, PDFs, databases…)
- Splitter: Intelligently slice long documents into chunks
- Embedding: Convert each chunk into a vector
- VectorStore: Store vectors, support similarity retrieval
- Retriever: Retrieve the most relevant chunks based on the user's question
- Augmented Prompt: Insert retrieval results into the Prompt
- Generation: The LLM generates an accurate answer based on the augmented Prompt
Mastering this process allows you to make an LLM understand any document, upgrading it from a "general-purpose chatbot" to a "domain knowledge expert".