Ingesting a 168-Chapter Novel into a Vector Database with LangChain and Milvus

RAG in Practice: 3 Steps to Store the Entire Demi-Gods and Semi-Devils in a Vector Database (Part 1)

Starting from scratch, use LangChain + Milvus to turn a 168-chapter novel into 3,042 vector chunks retrievable by AI.

Preface

I've been learning RAG (Retrieval-Augmented Generation) recently. I've read a lot of theory, but when I actually started building, I found more pitfalls than expected. So I used the e-book of Demi-Gods and Semi-Devils for practice, split the entire novel into 3,042 vector chunks, stored them in Milvus, and got the complete data ingestion pipeline running.

This article is suitable for Node.js developers who are getting started with RAG. After reading, you will master:

How to load EPUB e-books with LangChain and split them by chapter
The chunking principle and parameter tuning of RecursiveCharacterTextSplitter
Schema design and index construction for the Milvus vector database
Engineering practices for batch Embedding + insertion
The design concept of resumable processing (no fear of running out of tokens or network interruptions)

Complete runnable project code is attached at the end; clone it, change the configuration, and you can run it.

Project Overview

The goal of the entire project is: to turn Demi-Gods and Semi-Devils into a book that AI can "read," and then ask it questions.

This article focuses on the first part—the data ingestion pipeline:

Demi-Gods and Semi-Devils.epub
    │
    ▼
EPubLoader (load by chapter, 168 chapters)
    │
    ▼
RecursiveCharacterTextSplitter (split each chapter into smaller chunks)
    │
    ▼
OpenAIEmbeddings (each chunk → 1024-dimensional vector)
    │
    ▼
Milvus Vector Database (store + build index)

Core dependencies:

Package	Role
`@langchain/community`	EPubLoader document loader
`@langchain/textsplitters`	Text splitter
`@langchain/openai`	Embedding model calls (compatible with OpenAI API)
`@zilliz/milvus2-sdk-node`	Milvus vector database client

Step 1: Load the EPUB, Split into 168 Chapters

EPubLoader: One Line of Code for Chapter Splitting

The traditional approach is to manually parse the EPUB (essentially a ZIP package), extract HTML, clean tags... it's very tedious.

LangChain's EPubLoader does it all for you:

import { EPubLoader } from '@langchain/community/document_loaders/fs/epub';

const loader = new EPubLoader('./Demi-Gods and Semi-Devils.epub', {
    splitChapters: true,  // Key config: split into multiple Documents by chapter
});
const documents = await loader.load();
console.log(`Loading complete, total ${documents.length} chapters`);
// Output: Loading complete, total 168 chapters

Parameter	Role
`'./Demi-Gods and Semi-Devils.epub'`	EPUB file path
`splitChapters: true`	Split by chapter, each chapter becomes an independent `Document` object

The structure of each Document:

{
  pageContent: "Duan Yu was still tracing characters in the air, feeling sorry for himself...",  // Chapter body
  metadata: { chapter: 114, title: "..." }            // Chapter metadata
}

⚠️ Pitfall Record 1: The class name is EPubLoader (uppercase P), not EpubLoader. The new version of @langchain/community changed the naming; many old tutorials use lowercase p, and copying them directly will report does not provide an export named 'EpubLoader'.

Why is each chapter just one Document?

Chapter titles in novel EPUBs (like "Chapter 1: A Bold Journey on a Perilous Peak") are natural semantic boundaries. splitChapters: true makes the loader split along these boundaries. But a chapter might have thousands of characters, and direct embedding doesn't work well—the information is too mixed, and retrieval accuracy drops. So a second step is needed.

Step 2: Further Splitting with RecursiveCharacterTextSplitter

Why is further splitting needed?

Embedding models have input length limits. More importantly, text that is too long contains too much diverse information. Using "What martial arts does Duan Yu know?" to retrieve a 3,000-character chapter will dilute the similarity score with a large amount of irrelevant content.

So each chapter needs to be split into smaller chunks, allowing each chunk to focus on a relatively independent semantic unit.

How RecursiveCharacterTextSplitter Works

Its core idea is recursive splitting by priority:

Default separator priority: \n\n → \n → space → empty string

First split by double newlines (paragraphs). If a segment is still too long, split by single newlines. If still not enough, split by spaces... until each segment is within chunkSize.

import { RecursiveCharacterTextSplitter } from '@langchain/textsplitters';

const textSplitter = new RecursiveCharacterTextSplitter({
    separator: '\n',    // Specify separator (splitting by newline is natural for novels)
    chunkSize: 500,     // Max 500 characters per chunk
    chunkOverlap: 50,   // Adjacent chunks overlap by 50 characters to maintain context coherence
});

Parameter	Value	Why this setting
`separator`	`'\n'`	Novels break sentences by newlines; splitting by sentence is more reasonable than the default space splitting
`chunkSize`	`500`	Roughly equals a complete plot segment; too long makes retrieval inaccurate, too short fragments information
`chunkOverlap`	`50`	Prevents key information from being stuck right at the boundary between two chunks, making it unretrievable

Why is chunkOverlap important?

Suppose characters 499-501 happen to be "Duan Yu used the Wave-like Subtle Steps". Without overlap:

Chunk A: ...Duan Yu used the Wave-like      ← Searching "Wave-like Subtle Steps" won't match
Chunk B: Subtle Steps, his figure drifting...    ← Searching "Duan Yu used" won't match

With a 50-character overlap, this key information exists intact in at least one chunk, preventing loss due to "cutting in the middle of a keyword."

Step 3: Embedding + Milvus Ingestion

This is the most core step, divided into four parts.

3.1 Initializing the Embedding Model

import { OpenAIEmbeddings } from '@langchain/openai';

const embeddings = new OpenAIEmbeddings({
    apiKey: process.env.VITE_QWEN_API_KEY,
    model: 'text-embedding-v3',           // Qwen embedding model
    configuration: {
        baseURL: 'https://dashscope.aliyuncs.com/compatible-mode/v1',
    },
    dimensions: 1024,                      // Output 1024-dimensional vectors
});

Using @langchain/openai's OpenAIEmbeddings instead of the Qwen native SDK—because Qwen's embedding API is compatible with the OpenAI format, one set of code can adapt to multiple model providers.

⚠️ Pitfall Record 2: The value of configuration.baseURL must match the variable name in the .env file. The code writes process.env.VITE_QWEN_BASE_URL but the .env file calls it VITE_QWEN_API_URL; getting undefined sends the request to the wrong address, directly causing a Request timed out.

3.2 Designing the Milvus Collection Schema

A Milvus Collection is equivalent to a table in a relational database and requires a pre-defined field structure:

await client.createCollection({
    collection_name: 'ebook2',
    fields: [
        { name: 'id',          data_type: DataType.VarChar,
          max_length: 100, is_primary_key: true },
        { name: 'book_id',     data_type: DataType.VarChar,
          max_length: 100 },
        { name: 'book_name',   data_type: DataType.VarChar,
          max_length: 200 },
        { name: 'chapter_num', data_type: DataType.Int32 },
        { name: 'index',       data_type: DataType.Int32 },
        { name: 'content',     data_type: DataType.VarChar,
          max_length: 10000 },
        { name: 'vector',      data_type: DataType.FloatVector,
          dim: 1024 },
    ],
});

Field	Type	Why it's needed
`id`	VarChar	Primary key, format `01_114_28` (bookID_chapter_chunkIndex)
`book_id`	VarChar	Distinguishes different books, supports filtering searches by book
`book_name`	VarChar	Book name, convenient for display
`chapter_num`	Int32	Chapter number, supports filtering by chapter range
`index`	Int32	The chunk index within the chapter
`content`	VarChar	The original text of the chunk—the purpose of retrieval is to give this text to the LLM
`vector`	FloatVector(1024)	The vector representation of the text, the core retrieval field

⚠️ Pitfall Record 3: The content field is the easiest to overlook. Many people only store vectors and not the original text—but the "R" in RAG (Retrieval) is precisely to get the original text for the LLM to read. Not storing the original text makes the retrieval pointless.

3.3 Building the IVF_FLAT Index

await client.createIndex({
    collection_name: 'ebook2',
    field_name: 'vector',           // Build index on the vector field
    index_type: 'IVF_FLAT',         // Inverted File index
    metric_type: 'COSINE',          // Cosine similarity
    params: { nlist: 1024 },        // Number of K-Means cluster centers
});

Parameter	Meaning	Selection Basis
`IVF_FLAT`	Uses K-Means to cluster vectors into N clusters; during search, only the nearest few are checked	First choice for data volume < 100k records
`COSINE`	Cosine similarity	Standard choice for text semantic search
`nlist: 1024`	Cluster into 1024 clusters	Rule of thumb `4 × √N`, where N is the total data volume

3.4 Batch Embedding + Insertion

async function insertChunksBatch(chunks, bookId, chapterNum) {
    const insertData = await Promise.all(
        chunks.map(async (chunk, chunkIndex) => {
            const vector = await getEmbedding(chunk);  // Parallel API calls
            return {
                id: `${bookId}_${chapterNum}_${chunkIndex}`,
                book_id: bookId,
                book_name: BOOK_NAME,
                chapter_num: chapterNum,
                index: chunkIndex,
                content: chunk,
                vector: vector,
            };
        })
    );
    const result = await client.insert({
        collection_name: 'ebook2',
        data: insertData,
    });
    return Number(result.insert_cnt) || 0;
}

Key design: Promise.all for parallel embedding API calls. Chunks within the same chapter have no dependencies on each other; parallelism significantly speeds things up. If a chapter has 60 chunks, serial processing waits for 60 API round-trips, while parallel processing only waits for the slowest one.

Advanced: Resumable Processing

What if you run 168 chapters for the first time and run out of tokens at chapter 114? Re-running means calling the embedding API again—costly in money and time.

The solution: before each processing run, first check Milvus for which chapters have already been ingested:

async function getProcessedChapters(bookId) {
    const result = await client.query({
        collection_name: 'ebook2',
        filter: `book_id == "${bookId}"`,
        output_fields: ['chapter_num'],
        limit: 100000,
    });
    const chapters = result.data.map(r => r.chapter_num);
    return new Set(chapters);
}

Then skip them in the loop:

const processedChapters = await getProcessedChapters(bookId);

for (let i = 0; i < documents.length; i++) {
    const chapterNum = i + 1;
    if (processedChapters.has(chapterNum)) {
        console.log(`Chapter ${chapterNum} already processed, skipping`);
        continue;
    }
    // ... normal processing: split → embedding → insert
}

This way, no matter how many times it's interrupted, re-running continues from the breakpoint, and already ingested chapters won't be processed again. The Set data structure makes has() queries O(1), allowing millisecond-level checks for 168 chapters.

Execution Results

Loading complete, total 168 chapters
Processing chapter 1/168... split into 1 chunk... inserted 1 record
Processing chapter 2/168... split into 1 chunk... inserted 1 record
Processing chapter 12/168... split into 54 chunks... inserted 54 records
...
Processing chapter 168/168... split into 1 chunk... inserted 1 record

Total 3042 records inserted

All 168 chapters ingested, 3,042 vector chunks. The cloud Milvus now holds the entire "digital memory" of Demi-Gods and Semi-Devils.

Summary

This article covered three key steps for data ingestion plus one advanced technique:

EPubLoader — splitChapters: true loads EPUB by chapter in one line of code
RecursiveCharacterTextSplitter — Recursive splitting + chunkOverlap ensures retrieval accuracy
Milvus Schema + Index + Batch Insertion — Complete vector ingestion pipeline
Resumable Processing — client.query() first checks ingested chapters, Set.has() skips in milliseconds

The next article will cover the second part: how to use vector search to find relevant chunks, then use an LLM to generate answers based on the original text—truly enabling AI to "read" Demi-Gods and Semi-Devils.

📦 Complete runnable project code attached below 👇

Complete Project Code

Gitee Repository: https://gitee.com/dcx2758/ai_doubao_dcx/tree/main/ai/agent_in_action/tlbb/

File Structure

tlbb/
├── .env                    # Environment variable configuration
├── Demi-Gods and Semi-Devils.epub            # Novel file
├── src/
│   ├── main.mjs            # Data ingestion (content of this article)
│   ├── query.mjs           # Vector search
│   └── rag.mjs             # RAG Q&A
└── package.json

Quick Start

# 1. Clone repository
git clone [email protected]:dcx2758/ai_doubao_dcx.git
cd ai_doubao_dcx/ai/agent_in_action/tlbb

# 2. Install dependencies
pnpm install

# 3. Configure .env (fill in your API Key and Milvus address)
# VITE_QWEN_API_KEY=yourQwenAPIKey
# VITE_QWEN_EMBEDDING_MODEL=text-embedding-v3
# VITE_QWEN_API_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
# MILVUS_ADDRESS=yourMilvusAddress
# MILVUS_TOKEN=yourMilvusToken

# 4. Prepare the EPUB file and place it in the project root directory

# 5. Run ingestion
node src/main.mjs

Next: Let AI Read Demi-Gods and Semi-Devils: Complete Vector Search + LLM Intelligent Q&A Pipeline (Part 2)

What do you think of this "practical pitfall" style tutorial? Welcome to discuss in the comments 👏