Ingesting a 168-Chapter Novel into a Vector Database with LangChain and Milvus
RAG tutorials often stop at toy examples. This walkthrough surfaces the real-world friction points—API compatibility mismatches, chunk-boundary keyword loss, and token-budget interruptions—and shows concrete fixes that turn a demo into a resumable pipeline a developer can actually run against a full-length book.
A full Retrieval-Augmented Generation ingestion pipeline processes the 168-chapter novel *Demi-Gods and Semi-Devils* from EPUB to vector database. LangChain's EPubLoader splits the book by chapter, and a RecursiveCharacterTextSplitter further divides each chapter into 500-character chunks with a 50-character overlap to preserve keyword integrity at boundaries. The resulting 3,042 text fragments are embedded into 1,024-dimensional vectors using a Qwen model compatible with the OpenAI API format and stored in a Milvus collection with an IVF_FLAT index.
The pipeline uses Promise.all for parallel embedding calls within each chapter, cutting API wait time dramatically. A resumable processing design queries Milvus before each run to identify already-ingested chapters, storing them in a Set for O(1) lookups. This avoids re-processing and re-spending tokens after any interruption, whether from rate limits or network failures.
Schema design stores the original text alongside each vector, a detail often overlooked by beginners who save only vectors and defeat the purpose of retrieval. The index uses cosine similarity with 1,024 K-Means clusters, following the rule-of-thumb formula of four times the square root of total data volume.
Using LangChain's OpenAIEmbeddings class with a non-OpenAI provider (Qwen) works because the provider exposes an OpenAI-compatible endpoint. This pattern lets developers swap embedding backends without changing the LangChain abstraction layer, but it also creates a hidden coupling: the baseURL configuration must exactly match the environment variable name or the request silently fails.
Beginners frequently omit the content field in vector database schemas, storing only vectors. This mistake reveals a misunderstanding of RAG's retrieval step—the vector is the index, not the payload. The retrieved text is what the LLM consumes, so losing it breaks the entire pipeline.
The chunkOverlap parameter is not just a nice-to-have; without it, any key phrase that straddles a chunk boundary becomes invisible to similarity search. The example of '段誉使出凌波微步' split across two chunks demonstrates a failure mode that is deterministic and silent—no error is thrown, but recall drops.