跪拜 Guibai
← Back to the summary

Feeding a 1.5M-Character Martial-Arts Epic into a Vector Database, Chapter by Chapter

Table of Contents

  1. Overall Data Pipeline Overview
  2. EPubLoader: Loading E-books by Chapter
  3. RecursiveCharacterTextSplitter: Slicing Chapters into Small Chunks
  4. Batch Vectorization and Storage
  5. Summary

I. Overall Data Pipeline Overview

The first step of RAG is not retrieval, but turning raw documents into searchable vectors. The data pipeline requires four components working in relay: a Loader loads documents from files → a Splitter slices them into small chunks → an Embedding model converts them into vectors → Milvus stores them and builds an index.

image.png

II. EPubLoader: Loading E-books by Chapter

EPUB is a common e-book format, internally organized by chapters. LangChain's EPubLoader can directly parse EPUB files and output Document objects by chapter:

import { EPubLoader } from '@langchain/community/document_loaders/fs/epub';

const loader = new EPubLoader('./天龙八部.epub', {
    splitChapters: true   // Split by chapter, each chapter generates one Document
});

const documents = await loader.load();
console.log(`Loading complete, total ${documents.length} chapters`);

splitChapters: true is the key configuration—EPUB files have internal chapter markers, and the Loader splits the entire book into an independent Document array based on them. Each Document's pageContent is the full text of one chapter. Without this option, the entire book would be read as one massive Document, and subsequent splitting would lose chapter boundary information.

III. RecursiveCharacterTextSplitter: Slicing Chapters into Small Chunks

A chapter can contain thousands of characters. An Embedding interface cannot process overly long text in one go, and an LLM's context window cannot fit an entire chapter. A chapter needs to be sliced into multiple small chunks.

import { RecursiveCharacterTextSplitter } from '@langchain/textsplitters';

const textSplitter = new RecursiveCharacterTextSplitter({
    chunkSize: 500,      // Maximum 500 characters per chunk
    chunkOverlap: 50,    // Overlap of 50 characters between adjacent chunks
});

Two Core Parameters

Parameter Value Purpose
chunkSize 500 Maximum characters per chunk. Too large exceeds the embedding window, too small loses context.
chunkOverlap 50 Number of overlapping characters between adjacent chunks. Prevents critical information from being cut off exactly at a chunk boundary.

The significance of overlap: if a cut is made at the 250th character, "段誉施展凌波微步" might have "凌波微" in the previous chunk and "步" in the next—neither chunk's vector can fully represent the martial art name "凌波微步". A 50-character overlap ensures "凌波微步" appears completely in at least one chunk.

Splitting Chapter by Chapter

let totalInserted = 0;
const documentLen = documents.length;

for (let chapterIndex = 0; chapterIndex < documentLen; chapterIndex++) {
    const chapter = documents[chapterIndex];
    const chapterContent = chapter.pageContent;

    const chunks = await textSplitter.splitText(chapterContent);
    console.log(`Chapter ${chapterIndex + 1} split into ${chunks.length} chunks`);

    if (chunks.length === 0) {
        console.log(`Skipping empty chapter`);
        continue;
    }

    const insertedCount = await insertChunksBatch(
        chunks, bookId, chapterIndex + 1
    );
    totalInserted += insertedCount;
}

Processing chapter by chapter has three benefits: it preserves the chapter number field (allowing later retrieval results to show which chapter they came from), avoids loading the entire book into memory at once, and the chapter_num field on each chunk makes retrieval results more structured.

IV. Batch Vectorization and Storage

Collection Schema Design

await client.createCollection({
    collection_name: 'ebook',
    fields: [
        { name: 'id',           data_type: DataType.VarChar, max_length: 100, is_primary_key: true },
        { name: 'book_id',      data_type: DataType.VarChar, max_length: 100 },   // Which book
        { name: 'book_name',    data_type: DataType.VarChar, max_length: 200 },   // Book name
        { name: 'chapter_num',  data_type: DataType.Int32 },                      // Chapter number
        { name: 'index',        data_type: DataType.Int32 },                      // Chunk index within chapter
        { name: 'content',      data_type: DataType.VarChar, max_length: 10000 }, // Original text
        { name: 'vector',       data_type: DataType.FloatVector, dim: 1024 },     // Vector
    ]
});

image.png

Compared to the previous AI diary project, two structured fields—book_id and chapter_num—have been added. This allows later retrieval not only to get relevant text but also to pinpoint exactly which book and chapter the result comes from.

Batch Insertion

async function insertChunksBatch(chunks, bookId, chapterNum) {
    if (chunks.length === 0) return 0;

    const insertData = await Promise.all(
        chunks.map(async (chunk, chunkIndex) => {
            const vector = await getEmbedding(chunk);
            return {
                id: `${bookId}_${chapterNum}_${chunkIndex}`,
                book_id: bookId,
                book_name: BOOK_NAME,
                chapter_num: chapterNum,
                index: chunkIndex,
                content: chunk,
                vector: vector
            };
        })
    );

    const insertResult = await client.insert({
        collection_name: 'ebook',
        data: insertData
    });
    return Number(insertResult.insert_cnt) || 0;
}

Promise.all fires off all embedding requests for chunks within the same chapter in parallel. If a chapter has 20 chunks, 20 requests are sent simultaneously, and only after all vectors are obtained is a single client.insert() call made to write to the database—parallel requests save time, batch writes save network round trips.

Building an Index

await client.createIndex({
    collection_name: 'ebook',
    field_name: 'vector',
    index_type: IndexType.IVF_FLAT,
    metric_type: MetricType.COSINE,
    params: { nlist: 1024 }
});

nlist is the number of clusters for K-Means clustering—1024 clusters means that during a query, similarity is calculated only within the few clusters closest to the query vector, rather than against the entire dataset. For hundreds of thousands of vector records, an index is the prerequisite for millisecond-level response times.

V. Summary

  1. EPubLoader + splitChapters: Loads EPUB by chapter, turning each chapter into an independent Document. Preserves chapter boundaries, allowing later retrieval results to be traced back to specific chapters.
  2. RecursiveCharacterTextSplitter: chunkSize: 500 controls the character count per chunk, chunkOverlap: 50 prevents critical information from being cut off at boundaries. Splits chapter by chapter, skipping empty chapters.
  3. Multi-field Collection Design: Beyond vector and content, adding book_id, chapter_num, and index makes the data not only semantically searchable but also structurally queryable.
  4. Promise.all Parallelism + Batch Writes: Chunks within the same chapter request embeddings in parallel, with a single insert operation, balancing performance and reliability.
  5. IVF_FLAT + nlist: Clustered indexing reduces query complexity from O(n) full scan to O(log n) approximate cluster search.

The first part completes the data pipeline—from an epub file to a vector database. Next, searching Demi-Gods and Semi-Devils using natural language to see what snippets "What martial arts does Duan Yu know?" can retrieve.

—— A wise book with an assistant.