← Back to the summary

A Six-Step RAG Pipeline That Turns a Chinese Martial-Arts Epic into a Queryable AI Encyclopedia

Written at the beginning: Today's lesson was so interesting! The RAG we did before was just small-scale—manually writing a few stories about "Guangguang and Dongdong." Today, the teacher directly threw the EPUB e-book of "Demi-Gods and Semi-Devils" into the Milvus vector database! Loader loading → Splitter chunking → Embedding vectorization → Milvus storage → Semantic retrieval → LLM generated answer—a whole RAG pipeline was fully run through. Then I asked: "What martial arts does Duan Yu know?" The AI answered. I asked again: "What martial arts does Jiumozhi know?" The AI answered again. At that moment, I felt like I had an "AI-powered Jin Yong encyclopedia."

1. The Complete RAG Chain: From File to Answer

1.1 The "Five-Step Method" for Demi-Gods and Semi-Devils RAG

The teacher said:

"We have learned Loader, Splitter, Milvus, and the RAG process has been fully run through."

Demi-Gods and Semi-Devils.epub (e-book file)
  ↓ ① Loader (EPubLoader)
Array of Document objects (split by chapter)
  ↓ ② Splitter (RecursiveCharacterTextSplitter)
Small Document chunks (chunkSize=500)
  ↓ ③ Embedding (text-embedding-v3)
1024-dimensional vectors
  ↓ ④ Milvus Vector Database (storage + indexing)
Persistent storage, IVF_FLAT index acceleration
  ↓ ⑤ RAG Retrieval + Generation
Question → Vectorization → Search → Enhanced Prompt → LLM Answer

These five steps strung together form a complete, production-grade RAG process.

2. Step 1: EPubLoader Loads the E-book

2.1 What is EPUB?

EPUB is a standard format for e-books. Jin Yong's complete works and most web novels are in this format.

LangChain provides EPubLoader:

import { EPubLoader } from '@langchain/community/document_loaders/fs/epub';

const loader = new EPubLoader('./Demi-Gods and Semi-Devils.epub', {
    splitChapters: true,  // Split by chapter
});
const documents = await loader.load();
console.log(`Loading complete, total ${documents.length} chapters`);

splitChapters: true means automatically splitting the e-book into multiple Documents according to its chapters. One Document per chapter.

The teacher said:

"Loaders load documents from various sources. EPUB, CSV..., the corresponding Loader? LangChain has over 180 kinds."

3. Step 2: Text Splitting

3.1 RecursiveCharacterTextSplitter

import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";

const textSplitter = new RecursiveCharacterTextSplitter({
    chunkSize: 500,      // About 500 characters per chunk
    chunkOverlap: 50,    // Overlap 50 characters to maintain semantic coherence
});

Each chapter might have thousands of characters; you can't vectorize a whole chapter—it's too large for precise retrieval. You need to split it into semantically complete chunks.

The teacher said:

"Splitting symbols—。！？, chunk_size size, overlap overlap."

4. Step 3: Streaming Processing, Splitting and Vectorizing Simultaneously

4.1 Batch Processing by Chapter

for (let chapterIndex = 0; chapterIndex < documentLen; chapterIndex++) {
    const chapter = documents[chapterIndex];
    const chunks = await textSplitter.splitText(chapter.pageContent);
    console.log(`Split into ${chunks.length} data slices`);
    const insertedCount = await insertChunksBatch(chunks, bookId, chapterIndex + 1);
    totalInserted += insertedCount;
}

The teacher designed a "streaming processing" scheme—not loading all content into memory at once, but processing chapter by chapter. After each chapter is split, it is immediately vectorized and inserted into the database.

4.2 Vectorizing and Inserting

async function insertChunksBatch(chunks, bookId, chapterNum) {
    const insertData = await Promise.all(
        chunks.map(async (chunk, chunkIndex) => {
            const vector = await getEmbedding(chunk);
            return {
                id: `${bookId}_${chapterNum}_${chunkIndex}`,
                book_id: bookId,
                book_name: 'Demi-Gods and Semi-Devils',
                chapter_num: chapterNum,
                index: chunkIndex,
                content: chunk,
                vector: vector,
            };
        })
    );

    const insertResult = await client.insert({
        collection_name: COLLECTION_NAME,
        data: insertData,
    });

    return Number(insertResult.insert_cnt) || 0;
}

Each slice generates a 1024-dimensional vector before insertion, stored in Milvus together with the slice content.

5. Milvus Collection Design

5.1 Creating a Collection

await client.createCollection({
    collection_name: COLLECTION_NAME,
    fields: [
        { name: 'id', data_type: DataType.VarChar, max_length: 100, is_primary_key: true },
        { name: 'book_id', data_type: DataType.VarChar, max_length: 100 },
        { name: 'book_name', data_type: DataType.VarChar, max_length: 200 },
        { name: 'chapter_num', data_type: DataType.Int32 },
        { name: 'index', data_type: DataType.Int32 },
        { name: 'content', data_type: DataType.VarChar, max_length: 10000 },
        { name: 'vector', data_type: DataType.FloatVector, dim: VECTOR_DIM },
    ]
});

This collection design contains all the necessary information:

Field	Purpose	Example
`id`	Unique identifier	`1_1_0` (Chapter 1, Chunk 0)
`book_id`	Book number	Supports multiple books
`book_name`	Book title	Demi-Gods and Semi-Devils
`chapter_num`	Chapter number	Chapter 3
`index`	Slice sequence number	Chunk 5
`content`	Original text content	"Duan Yu followed Zhong Ling..."
`vector`	1024-dim semantic vector	Used for similarity search

5.2 Creating an Index

await client.createIndex({
    collection_name: COLLECTION_NAME,
    field_name: 'vector',
    index_type: IndexType.IVF_FLAT,
    metric_type: MetricType.COSINE,
    params: { nlist: 1024 }
});

The teacher said:

"IVF_FLAT—clustered index, millisecond level. COSINE—high-dimensional similarity, not slow even with large data volumes."

6. RAG Retrieval and Generation

6.1 Searching for Relevant Snippets

async function retrieveRelevantContent(question, k = 3) {
    const queryVector = await getEmbeddings(question);
    const searchResult = await client.search({
        collection_name: COLLECTION_NAME,
        vector: queryVector,
        limit: k,
        metric_type: MetricType.COSINE,
        output_fields: ['id', 'book_id', 'chapter_num', 'index', 'content'],
    });
    return searchResult.results;
}

6.2 Enhanced Prompt + Generation

const prompt = `You are a professional assistant for the novel "Demi-Gods and Semi-Devils".
Answer questions based on the novel, using accurate and detailed language.
Please answer the question based on the following novel excerpt content:
${context}

User question: ${question}

Answer requirements:
1. If there is relevant information in the excerpt, please provide a detailed and accurate answer based on the novel content.
2. You can synthesize content from multiple excerpts to provide a complete answer.
3. If there is no relevant information in the excerpts, please honestly tell the user.
4. The answer must be accurate, conforming to the novel's plot and character settings.
5. You can quote original text to support your answer.
AI Assistant's Answer:`;

6.3 Actual Q&A Results

User asks: "What martial arts does Duan Yu know?"

Relevant snippets found (sorted by cosine similarity):

1. [Score: 0.8921] Chapter: Chapter 3
   Duan Yu began to practice "Northern Divine Art" as shown in the diagram...
2. [Score: 0.8745] Chapter: Chapter 5
   Duan Yu used "Wave-like Subtle Steps," his footwork erratic...
3. [Score: 0.8312] Chapter: Chapter 7
   Duan Yu was anxious, and the Six Meridians Divine Sword involuntarily...

AI's Answer:

"Duan Yu possesses several peerless martial arts. He learned the Northern Divine Art from the jade wall in Mount Wuliang, which allows him to absorb others' internal energy for his own use; later, he learned the Six Meridians Divine Sword at Tianlong Temple, using fingers as swords with infinite power; in addition, he is proficient in Wave-like Subtle Steps, an extremely exquisite lightness skill with erratic footwork and unpredictable changes..."

This is the power of RAG—the AI doesn't "recite" knowledge from thin air, but retrieves relevant content from the vector library and generates an answer based on the original text.

7. The Teacher's "Demi-Gods and Semi-Devils RAG" Summary

The teacher said:

"Loader loads documents from various sources—EPUB, CSV... the corresponding Loader. Splitter chunks—Separator splitting symbols 。！？, chunk_size, overlap. Embedding—1024 dimensions → millions of words. Milvus database. RAG—Cosine, top_k."

These five modules strung together form a complete, scalable RAG knowledge base system. Want to switch books? Just change the file name. Want to switch knowledge base categories? Just change the Loader.

8. Summary: Complete RAG Chain Overview

Step	Technology	Core Parameters	What it does
① Load	`EPubLoader`	`splitChapters: true`	Reads e-book, splits by chapter
② Split	`RecursiveCharacterTextSplitter`	`chunkSize: 500`, `overlap: 50`	Splits into semantically complete chunks
③ Vectorize	`OpenAIEmbeddings`	`dimensions: 1024`	Text → 1024-dim vector
④ Store	`MilvusClient.insert`	`IVF_FLAT` index, `COSINE` similarity	Stores in vector database
⑤ Retrieve	`client.search`	`limit: 3`, `COSINE`	Semantic search for most relevant snippets
⑥ Generate	`ChatOpenAI.invoke`	`temperature: 0.1`	Generates answer based on original text

From file to answer, six steps completed. This is the knowledge base tech stack for modern AI applications.

Written at the End

Today's lesson was really interesting! I personally fed "Demi-Gods and Semi-Devils" to the AI, and then asked it about martial arts, characters, and plots, and it could answer all of them. What excites me even more is that this entire process—Loader → Splitter → Embedding → Milvus → RAG—is universal.

Change the Loader to handle PDFs, change the file to handle Jin Yong's complete works. This "knowledge base pipeline" can be applied to any scenario requiring "AI + private knowledge."

Next time an interviewer asks you: "What is the complete RAG process in a production environment?"

You can calmly say:

"The complete RAG process in a production environment is six steps. ① Loader—Use EPubLoader, PDFLoader, etc. to load content from files and convert it into standard Documents; ② Splitter—Use RecursiveCharacterTextSplitter to split by delimiters like periods into semantically complete chunks, setting chunkSize and chunkOverlap; ③ Embedding—Use an Embedding model to convert each chunk into a high-dimensional vector (e.g., 1024 dimensions); ④ Storage—Store the vectors and original text into a vector database like Milvus, building an IVF_FLAT index and COSINE similarity metric; ⑤ Retrieval—When a user asks a question, vectorize the question and use client.search to find the top K most relevant snippets in Milvus; ⑥ Generation—Assemble the retrieved snippets into an enhanced Prompt and hand it to the LLM to generate the final answer. These six steps strung together form a complete RAG knowledge base system."

Then look at the interviewer's satisfied expression and silently think: This round, nailed it again.

All code examples in this article are from classroom learning materials and are genuinely runnable.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

Darling噜啦啦

[Strong][Strong][Strong]