跪拜 Guibai
← Back to the summary

From URL to Answer: The Full RAG Pipeline for a Single Web Page

The previous two articles have already broken down the basic flow of RAG, as well as what problems Document, Chunk, metadata, and vector databases solve.

However, the previous examples had an obvious idealized condition: the Documents participating in the retrieval were created manually.

A real knowledge base usually faces not a few pre-organized strings, but raw materials like web pages, PDFs, Word documents, and Markdown.

If the knowledge source is a web page, the system must at least complete the following chain:

Web Page URL
→ Download HTML
→ Extract main content
→ Convert to Document
→ Split into Chunks
→ Generate Embeddings
→ Write to vector database
→ Retrieve relevant fragments based on a question
→ Pass fragments to the large model for answering

This article uses a real web page as the knowledge source to fully implement web page loading, document splitting, vectorization, similarity search, and model answering.

Prepare Dependencies

The example uses LangChain.js, Cheerio, a Tongyi Qianwen-compatible interface, and an in-memory vector store:

pnpm add \
  @langchain/community \
  @langchain/textsplitters \
  @langchain/classic \
  @langchain/openai \
  cheerio \
  dotenv

Step 1: Use a Loader to Turn a Web Page into a Document

Different types of knowledge sources require different loading methods.

Web Page → Web Loader
PDF      → PDF Loader
Word     → Docx Loader
CSV      → CSV Loader

The Loader's responsibility is not to generate vectors, but to uniformly convert raw content in different formats into LangChain Documents.

This article uses CheerioWebBaseLoader to load a web page:

import { CheerioWebBaseLoader } from 
  '@langchain/community/document_loaders/web/cheerio';

const sourceUrl = 'https://juejin.cn/post/7660707431753678854';

const cheerioLoader = new CheerioWebBaseLoader(
  sourceUrl,
  {
    selector: '.article-viewer > :not(style)',
  },
);

const documents = await cheerioLoader.load();

After loading, you get an array of Documents.

Within it:

pageContent: The main content of the web page extracted via the CSS selector
metadata.source: The original web page address
metadata.title: The web page title

At this point, no Embeddings have been generated, and no retrieval has been performed. The Loader has only completed the conversion from "raw web page to standard document."

Use a CSS Selector to Extract the Complete Main Content

Web page main content usually contains more than just paragraphs; it also includes:

Therefore, the CSS selector needs to cover multiple content elements within the main content container. The current web page uses the following selector, reading the direct child elements of the main content container and excluding style tags:

selector: '.article-viewer > :not(style)'

This allows the main text, headings, lists, tables, and code to be saved uniformly into pageContent, providing complete original content for subsequent splitting and retrieval.

The CSS selector here depends on the target website's current DOM structure. If the website is redesigned, the selector must be adjusted accordingly. A formal project also needs to check the scraping results, handle empty content, and comply with the target website's usage rules.

Step 2: Recursively Split Long Documents

After the web page main content is loaded, what you get is still a whole long document.

Directly converting the entire article into a single vector would mix multiple topics. When a user asks only a local question, the retrieval granularity would be too coarse.

Therefore, you need to use RecursiveCharacterTextSplitter for splitting:

import { RecursiveCharacterTextSplitter }
  from '@langchain/textsplitters';

const textSplitter = new RecursiveCharacterTextSplitter({
  chunkSize: 400,
  chunkOverlap: 80,
  separators: [
    '\n\n', '\n',
    '。', '！', '？', '；', '，',
    ' ', ''
  ],
});

const splitDocuments =
  await textSplitter.splitDocuments(documents);

On the current web page, this step splits 1 original Document into 30 Chunks.

The specific number will change depending on the web page content and splitting configuration; it is not a fixed result.

The Recursive Splitting Strategy of Separators

RecursiveCharacterTextSplitter tries progressively according to the separator order:

Try paragraphs first
→ If the paragraph is still too long, try newlines
→ If still too long, try periods
→ Then try exclamation marks, question marks, semicolons, commas
→ Finally, fall back to spaces or characters

The goal of this approach is to prioritize preserving larger natural semantic boundaries while trying to make each Chunk close to chunkSize.

Use an Empty String for Character-Level Fallback

The separator array keeps an empty string '' at the end for character-level splitting. Even when encountering code, menus, ancient texts, or long texts without punctuation, the splitter can still control the content to be close to the chunkSize range.

What Problem Does chunkOverlap Solve?

Suppose a piece of important information happens to be at the boundary of two Chunks:

Chunk 1: ...Node.js's main thread is not suitable for executing time-consuming synchronous I/O
Chunk 2: Therefore, asynchronous file reading should be prioritized in web services...

When the two fragments appear alone, they may both lack the complete causal relationship.

chunkOverlap: 80 will make adjacent Chunks retain a portion of overlapping content, reducing the probability of boundaries cutting off semantics.

But overlap is not the bigger the better. Too much overlap will create a large number of duplicate vectors, increasing storage, Embedding, and retrieval costs.

There is no fixed answer for chunkSize and chunkOverlap applicable to all projects; they need to be evaluated in combination with the document structure, Embedding model, and actual retrieval results.

Step 3: Generate Embeddings in Batches

The example calls DashScope through an OpenAI-compatible interface:

import { OpenAIEmbeddings } from '@langchain/openai';

const embeddings = new OpenAIEmbeddings({
  apiKey: process.env.DASHSCOPE_API_KEY,
  model: process.env.EMBEDDINGS_MODEL_NAME,
  batchSize: 10,
  configuration: {
    baseURL: process.env.DASHSCOPE_BASE_URL,
  },
});

The batchSize: 10 here controls how many texts are submitted in a single API request. The current example generated 30 Chunks, and LangChain will automatically complete multiple requests in batches of 10.

When using an OpenAI-compatible interface, the model name, batch size, vector dimensions, and request limits still need to be configured according to the actual service provider's interface requirements.

Step 4: Write to an In-Memory Vector Database

After splitting is complete, you can create an in-memory vector store directly from the Documents:

import { MemoryVectorStore }
  from '@langchain/classic/vectorstores/memory';

const vectorStore = await MemoryVectorStore.fromDocuments(
  splitDocuments,
  embeddings,
);

fromDocuments() internally completes two things:

Read each Document.pageContent
→ Call the Embedding Model to generate vectors
→ Save the vectors, pageContent, and metadata

MemoryVectorStore is suitable for teaching, testing, and small demos. Its data is only saved in the current process's memory and will disappear after the program restarts, making it unsuitable as a persistence solution for a formal knowledge base.

Step 5: Perform Similarity Search with Scores

The user's question is still a plain string:

const question = 'What are the common APIs of the fs module?';

Using similaritySearchWithScore(), you can directly pass in the question string and return the most relevant documents and their scores:

const scoredResults =
  await vectorStore.similaritySearchWithScore(
    question,
    3,
  );

This method will first call the Embedding Model to generate a question vector, and then compare it with the document vectors in the vector store.

Reading Similarity Scores

The currently used MemoryVectorStore calculates cosine similarity by default; a larger value indicates higher relevance.

The running result is similar to:

[Document 1] Cosine Similarity: 0.6843
[Document 2] Cosine Similarity: 0.6454
[Document 3] Cosine Similarity: 0.6131

Different vector databases may return similarity or distance. The current MemoryVectorStore returns cosine similarity, where a larger value means more relevant; when switching to other vector databases, scores should be interpreted according to the corresponding implementation.

Step 6: Add Retrieval Results to the Prompt

Vector retrieval returns relevant Documents, but the generation model has not been called yet.

First, compose the Top 3 fragments into context:

const docs = scoredResults.map(([doc]) => doc);

const context = docs
  .map(
    (doc, index) =>
      `[Fragment ${index + 1}]\n${doc.pageContent}`,
  )
  .join('\n\n-----\n\n');

Then pass the context and the question together to the model:

const prompt = `
You are an article reading assistant. Please answer questions based only on the given article fragments;
If the fragments do not provide an answer, please clearly state that the information is insufficient.

Article Fragments:
${context}

Question: ${question}

Answer:
`;

const response = await model.invoke(prompt);
console.log(response.content);

After fixing the web page selector, the retrieved fragments contain the code and lists from the article, and the model can identify content like readFileSync, readFile, and node:fs/promises based on the context.

The complete data flow is:

Juejin Web Page
→ CheerioWebBaseLoader
→ Original Document
→ RecursiveCharacterTextSplitter
→ 30 Chunks
→ OpenAIEmbeddings
→ MemoryVectorStore
→ similaritySearchWithScore
→ Top 3 Documents
→ Prompt
→ Chat Model Answer

What Gaps Does This Demo Have from a Production Environment?

The current example has already run through the RAG chain, but it is not yet a production-grade knowledge base.

At least the following still need to be addressed:

In particular, web page content cannot be considered trusted instructions by default. A RAG system should treat retrieval results as reference materials, rather than allowing web page text to override system rules.

Summary

Integrating a web page into RAG is not just "requesting a URL" and being done with it.

The Loader is responsible for converting the web page main content into a standard Document, the Splitter generates Chunks according to semantic boundaries, the Embedding Model converts text into vectors, the Vector Store saves and retrieves relevant content, and finally the generation model answers questions based on the retrieved fragments.

The complete process can be summarized as:

URL
→ CheerioWebBaseLoader
→ Document
→ RecursiveCharacterTextSplitter
→ Embedding
→ MemoryVectorStore
→ Top K Retrieval
→ Prompt
→ Model Answer

After passing through this data pipeline, the web page content truly transforms from an external resource into RAG knowledge that can participate in semantic retrieval.