Ingesting a 1.2M-Word Novel into a Vector DB for RAG, Step by Step
RAG pipelines look trivial in demos but break at scale on details like chunk boundaries, dimension mismatch, and type coercion. This walkthrough surfaces those hidden failure points and shows how a few defensive patterns keep a real-world ingestion job from silently corrupting its own data.
A full ingestion-to-answer pipeline processes the classic wuxia novel "Tian Long Ba Bu" into a vector database. The EPUB is split into 50 chapters, chunked into roughly 2,400 overlapping text segments, and embedded into 1,024-dimensional vectors using Alibaba's text-embedding-v3 model via an OpenAI-compatible endpoint. All vectors land in a Milvus collection indexed with IVF_FLAT and cosine similarity.
A separate retrieval module performs pure vector search, while the RAG module assembles a prompt from the top-k fragments and feeds it to a Chat model to generate natural-language answers about characters' martial arts skills. The design emphasizes defensive coding: parallel embedding with Promise.all, primary keys that encode business semantics, and return-value normalization to prevent NaN propagation.
The stack uses LangChain's community loaders and splitters, but the real substance is in the engineering decisions—why 1024 dimensions instead of 1536, why nlist=1024, and how a 50-character overlap prevents context from falling on chunk boundaries.
Most RAG tutorials stop at "search + LLM," but the real engineering lives in the ingestion pipeline: chunk strategy, dimension tuning, index parameters, and type-safety at the database boundary.
The project treats the OpenAI API format as a universal interface layer, decoupling the embedding model, chat model, and vector database so any component can be swapped by changing environment variables alone.
RecursiveCharacterTextSplitter's recursive degradation from semantic boundaries (paragraphs, sentences) down to hard character cuts is a practical compromise that avoids the worst fragmentation without requiring a custom splitter.
Parallel embedding with Promise.all is safe here because each chunk's API call is stateless, but the article correctly notes that rate limits become the real ceiling at higher volumes.
The prompt's five constraint instructions are not boilerplate; each one counters a specific RAG failure mode: vagueness, myopic focus on the first fragment, hallucination, contradiction with source material, and unsupported claims.