Artificial Intelligence

From URL to Answer: The Full RAG Pipeline for a Single Web Page

By 东风破_ · Jul 19, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

The gap between a working RAG demo and a production knowledge base is wide and filled with unglamorous failure modes — selector breakage, empty pages, duplicate content, and prompt injection — that most tutorials skip. This walkthrough names them explicitly, giving a team a concrete checklist of what to harden before shipping.

Summary

A real Juejin article serves as the knowledge source for a full Retrieval-Augmented Generation pipeline built with LangChain.js. The process starts with CheerioWebBaseLoader and a CSS selector to extract clean page content into a Document, then uses RecursiveCharacterTextSplitter with Chinese punctuation separators to break it into 30 overlapping chunks. Embeddings are generated in batches via a DashScope-compatible API and stored in an in-memory vector store for cosine similarity search. The top three retrieved fragments are fed into a prompt that instructs the model to answer only from the provided context, completing the URL-to-answer chain. The piece closes by listing the gaps between this demo and a production system: selector fragility, empty-content handling, incremental sync, persistent storage, similarity thresholds, metadata filtering, reranking, and prompt injection risks.

Takeaways

— CheerioWebBaseLoader with a CSS selector converts a live web page into a LangChain Document with pageContent and metadata in one step.

— RecursiveCharacterTextSplitter with a separator list of `\n\n`, `\n`, Chinese punctuation, space, and empty string preserves semantic boundaries while enforcing chunk size.

— chunkOverlap of 80 characters prevents causal relationships from being severed at chunk boundaries, but excessive overlap creates duplicate vectors and raises costs.

— MemoryVectorStore.fromDocuments() handles embedding generation and storage in one call, but data vanishes on restart — it is only for demos and testing.

— similaritySearchWithScore() returns cosine similarity scores where higher values mean greater relevance; other vector databases may return distance metrics instead.

— The final prompt explicitly constrains the model to answer only from provided fragments and to state when information is insufficient, a basic guard against hallucination.

— Production hardening requires handling selector failures, empty or duplicate content, incremental document sync, persistent vector storage, similarity thresholds, metadata filtering, reranking, and prompt injection from untrusted web text.

Conclusions

Most RAG tutorials stop at the happy path; this one enumerates the failure modes — selector drift, empty pages, duplicate content, prompt injection — that turn a demo into a maintenance burden.

The separator list is the unsung hero of chunking quality. Prioritizing paragraph breaks and Chinese punctuation before falling back to character-level splitting is what keeps code blocks and lists from being shredded mid-structure.

Treating retrieved web content as reference material rather than trusted instructions is a security posture that many early RAG systems overlook, and it becomes critical the moment a knowledge base ingests arbitrary URLs.

Concepts & terms

RecursiveCharacterTextSplitter

A LangChain text splitter that tries to break text at increasingly granular separators (paragraphs, newlines, sentences, words, characters) to keep chunks near a target size while preserving the largest possible semantic units.

chunkOverlap

The number of characters shared between adjacent chunks. It prevents information that straddles a chunk boundary from losing context, at the cost of some storage and embedding duplication.

MemoryVectorStore

An in-memory vector database that stores embeddings in the current process. It disappears on restart and is suitable only for prototyping, testing, and small demos.

cosine similarity

A metric that measures the angle between two vectors. In RAG retrieval, a higher cosine similarity score indicates a document chunk is semantically closer to the query.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗