Feeding a 1.5M-Character Martial-Arts Epic into a Vector Database, Chapter by Chapter
RAG demos often skip the ingestion plumbing, but chunk boundaries, overlap, and metadata schema are where real retrieval quality is won or lost. A 50-character overlap is the difference between finding "凌波微步" and missing it entirely, and chapter-level metadata turns a semantic haystack into a structured, citable knowledge base.
The pipeline starts with LangChain's EPubLoader, which parses the novel's internal chapter markers to produce one Document per chapter instead of one monolithic blob. A RecursiveCharacterTextSplitter then breaks each chapter into 500-character chunks, with a 50-character overlap that prevents martial-arts move names from being severed across chunk boundaries. Each chunk is sent to an embedding API in parallel via Promise.all, and the resulting 1024-dimension vectors land in a Milvus collection that also stores book ID, chapter number, and chunk index.
An IVF_FLAT index with 1,024 clusters makes similarity search sub-linear, so queries scan only nearby clusters rather than the entire dataset. The collection schema deliberately carries structured metadata alongside the vectors, so a retrieval result can report not just relevant text but its exact chapter location. The entire ingestion runs chapter by chapter to keep memory usage low and to preserve chapter-level provenance on every chunk.
Chunk overlap is often treated as a tuning knob, but for domain-specific text like wuxia novels where multi-character named entities are dense, it becomes a hard correctness requirement — without it, martial-arts move names simply disappear from the index.
The decision to embed chapter-level metadata directly into the vector database schema, rather than keeping it in a separate relational store, means a single query can return both semantic matches and their exact book location without a join.
Running embedding requests in parallel per chapter is a pragmatic middle ground: it avoids the latency of sequential calls without the complexity of full-streaming backpressure that a book-wide fan-out would require.