跪拜 Guibai
← Back to the summary

Adding AI Q&A to a Docusaurus Knowledge Base with RAG and SSE Streaming

Actually, my needs are quite simple. The plain-text search on my knowledge base site, which was based on Algolia, had become a bit outdated. I wanted to integrate AI Q&A that could simultaneously retrieve content from multiple places and provide a conversational summary, rather than simply returning a list of articles containing text strings. (Algolia later launched NeuralSearch, but it requires an additional activation and payment.)

As a frontend developer, I've been looking for an entry point to explore AI Agent-related technologies. Integrating RAG (Retrieval-Augmented Generation)-based AI Q&A into my knowledge base is undoubtedly the best practice project for getting hands-on with Prompt Engineering and RAG technology.

How to interact? Since the original Algolia search window was in the upper right corner, I simply added an AI Q&A button right next to it to trigger a dialogue window. The effect is as follows:

ai q&a dialogue window frontend effect

Online Experience Link. Click 'Ask AI' in the upper right corner to try it out.

This is a bit like a customer service assistant — speaking of which, a customer service assistant can be considered the best project for getting started with AI Agents. You design multiple reply prompts based on questions, call API methods or query databases through Function Calling. My first practice project was building a customer service assistant, which helped me get familiar with the basic system of AI Agents.

Next, I needed a lightweight backend service to implement the RAG service interface and connect to a vector database.

Overall Design and Technology Selection

For full-stack development, the first step is to decide on the tech stack. My core requirements are lightweight, low-cost, and maintenance-free:

Key Engineering Issues

The following modules are laid out exactly according to the RAG data flow sequence: Offline Data Production (Chunking -> Ingestion) -> Online Service Interface (Prompt Assembly -> LLM Streaming Passthrough) -> Frontend Display (Streaming Rendering).

1. Document Chunking

Chunking Strategy: Split text semantically based on heading bars and paragraphs.

We need to extract all Markdown (.md/.mdx) documents from the docs/ and blog/ directories in the Docusaurus project and perform reasonable chunking for subsequent vectorization (Embedding). Since Markdown itself is structured, H1/H2/H3 are natural semantic boundaries. Each chunk is a complete small topic, ready to be fed directly to the embedding model. Each chunk must retain routing information (sourceUrl) to facilitate displaying source links in the frontend Q&A.

document chunking Several libraries were used to assist with this process:

The entire pipeline:

.md source file
  → unified + remark-parse: Parse into AST (tree structure)
  → remark-frontmatter: Extract YAML from the header as metadata
  → Traverse AST to find H2/H3 nodes as chunking anchors
  → mdast-util-to-string: Convert the text content of chunk nodes to plain text
  → Output: { sourceUrl, title, content, ...metadata }

Why not do it in one step? Markdown seems simple, but the YAML header, code blocks, links, and images are all "structured information." You have to identify them all first to cut precisely, hierarchically, and without destroying the context.

Note: Only chunking down to H2/H3 is appropriate; any finer and it becomes fragmented. For particularly long sections, you can subdivide further by paragraph.

You can configure the scripts command in package.json so that the script can be executed within the GitHub Workflow process to trigger document chunking and automatically upload to the vector database:

"scripts": {
    "rag:parse": "tsx scripts/rag/test-parser.ts"
}

2. Text Vectorization and Incremental Ingestion

Using the text-embedding-3 model, the chunked text is converted into vector representations (turning "words" into "coordinates" to later find the nearest ones in the vector space).

embedding example

There are a few details when storing in ChromaDB:

text vectorization The actual data format after chunking is as follows:

{
  "content": "[Document Path: /docs/afreshjs/Node.js/Advanced Core Concepts | Section: Advanced Core Concepts > Advanced Core Concepts > 3. Underlying Principles of Modularization (CJS vs ESM) > 3.2 Circular Dependency]\n### 3.2 Circular Dependency\n\nWhen `a.js` references `b.js`, and `b.js` also references `a.js`:\n\n- **CommonJS Behavior**:\n  When loading a module, CJS first creates an empty object `module.exports` for that module in `require.cache`. When a circular reference occurs, `b.js` will get the **incomplete `exports` object of `a.js` that hasn't finished executing yet**. This can lead to runtime `undefined` errors.\n  _(CJS exports a copy/shallow copy of the value)_\n\n- **ESM (ECMAScript Modules) Behavior**:\n  ESM loading is divided into three phases: "parsing", "instantiation", and "evaluation". ESM exports are **Live Bindings**, meaning the exported variable and the variable inside the original module point to the same memory address.\n  Therefore, when handling circular references, as long as you don't immediately try to read that uninitialized variable, the engine can perfectly handle the module dependency graph.\n\n---",
  "metadata": {
    "sourceUrl": "/docs/afreshjs/Node.js/Advanced Core Concepts",
    "title": "Advanced Core Concepts",
    "h1": "Advanced Core Concepts",
    "h2": "3. Underlying Principles of Modularization (CJS vs ESM)",
    "h3": "3.2 Circular Dependency",
    "chunkIndex": 5
  }
}

Incremental Ingestion

There was a small hiccup during the first full ingestion: the data carried by the interface was too large, causing Nginx to throw a 413 (Request Entity Too Large) error. Besides adjusting the Nginx threshold (client_max_body_size), to be safe, multiple interfaces were used for batch uploads to avoid triggering interface timeouts (504).

Subsequent updates use incremental ingestion: Based on Git Diff, only changed files are re-embedded, avoiding the need for a full upload every time the knowledge base is modified. This logic is written in GitHub Actions.

ChromaDB's API design is very suitable for this scenario; the underlying HNSW index is self-maintaining and doesn't require manual rebuilding:

// Add new
await collection.add({ ids, documents, embeddings, metadatas });
// Update if exists, add if not (by id) — most commonly used for ingestion
await collection.upsert({ ids, documents, embeddings, metadatas });
// Update existing (will error if not found)
await collection.update({ ids, embeddings, metadatas });
// Delete
await collection.delete({ ids: [...] });

3. Prompt Design

For the AI Q&A responses, I designed the following Prompt to strictly limit the AI's behavior and avoid hallucinations:

# Role

You are a knowledge base assistant. You can only answer user questions based on the 'Reference Documents'. Do not fabricate.

# Response Requirements

1. First, directly answer the user's question in 1-2 sentences.
2. Then, supplement with key details (from the reference documents).
3. Finally, list the cited sources (only list the document name + section, do not paste long URL links).
4. If the reference documents are insufficient to answer, clearly state "The knowledge base does not have relevant content."

# Reference Documents

${retrieved_chunks}

# User Question

${user_query}

4. Streaming Q&A (SSE) and Nginx Proxy Pitfalls

This is a typical application scenario for SSE: the user inputs a question, the backend retrieves the most relevant data from the database and feeds it to the model, then processes the model's returned result into a streaming response, which the frontend receives and displays to the user in real-time (typewriter effect).

To enable an SSE interface, three things are indispensable: Response Headers + Backend Implementation + Nginx Reverse Proxy Configuration.

1. Response Headers

content-type: text/event-stream
cache-control: no-cache          # Prevent intermediate proxy caching
connection: keep-alive           # Maintain a persistent connection
x-accel-buffering: no            # Tell Nginx not to buffer (critical)

2. Backend Implementation (Hono) Using streamSSE from hono/streaming (which wraps the backend response into SSE format data: ...\n\n), the upstream stream: true allows DeepSeek to output as it thinks. The core is "passthrough of the upstream stream":

import { Hono } from "hono";
import { streamSSE } from "hono/streaming";

const app = new Hono();

app.get("/rag-api/ask", async (c) => {
  return streamSSE(c, async (stream) => {
    // 1. Request upstream LLM, enable streaming
    const upstream = await fetch("https://api.deepseek.com/chat/completions", {
      method: "POST",
      headers: {
        /* ... */
      },
      body: JSON.stringify({ /* ... */ stream: true }),
    });

    if (!upstream.body) return;

    // 2. Passthrough upstream stream: read → parse → writeSSE
    const reader = upstream.body.getReader();
    const decoder = new TextDecoder();
    let buffer = "";

    // Listen for client disconnection, clean up upstream (don't waste tokens)
    stream.onAbort(async () => {
      await reader.cancel();
    });

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      buffer += decoder.decode(value, { stream: true });
      const lines = buffer.split("\n");
      buffer = lines.pop() || ""; // The last line might be incomplete, save for next round

      for (const line of lines) {
        if (!line.startsWith("data: ")) continue;
        const data = line.slice(6).trim();
        if (data === "[DONE]") {
          await stream.writeSSE({ data: "[DONE]" });
          continue;
        }
        try {
          const json = JSON.parse(data);
          const content = json.choices[0]?.delta?.content || "";
          if (content)
            await stream.writeSSE({ data: JSON.stringify({ content }) });
        } catch (e) {
          console.error("SSE parsing error:", e);
        }
      }
    }
  });
});

3. Nginx Reverse Proxy Must Disable Buffering (The Most Common Pitfall) Nginx buffers responses by default, which can "freeze" SSE streams, making them invisible until the buffer is full and flushed all at once. You must add:

location /rag-api/ {
    proxy_pass http://127.0.0.1:3000/;
    proxy_buffering off;          # Critical! Disable buffering
    proxy_cache off;              # Disable caching
    proxy_http_version 1.1;       # SSE requires HTTP/1.1
    chunked_transfer_encoding on;
    proxy_read_timeout 60s;       # Prevent timeout disconnection
}

Summary of common pitfalls:

Pitfall Symptom Solution
Nginx default buffering Stream "freezes", outputs in a large chunk proxy_buffering off
Client disconnects but upstream still running Tokens continue to be consumed stream.onAbort + reader.cancel
Chunk spans multiple lines JSON.parse error Buffer accumulation + lines.pop()
EventSource doesn't support POST Inconvenient parameter passing Use @microsoft/fetch-event-source

Why use the @microsoft/fetch-event-source library for frontend SSE requests?

This library uses fetch under the hood to send requests, supports POST and custom Headers, and internally encapsulates ReadableStream compatibility handling. Business code only needs to care about the onmessage callback.

Summarizing the above flow:

DeepSeek API (ReadableStream)
    │ raw bytes (SSE format: data: {...}\n\n)
    ▼
getReader() async iteration
    │
    ▼
TextDecoder → buffer accumulation → split by \n
    │
    ▼
JSON.parse → extract content
    │
    ▼
writeSSE({ data: JSON.stringify({content}) })
    │
    ▼
Hono internally writes to response.body (writable)
    │
    ▼
Nginx no buffering → Client immediately visible

Evaluating Model Response Quality

After RAG is up and running, how do you know if it "answers well"? Relying on manual spot-checks is obviously unreliable. A relatively simple evaluation process is used here:

1. Establish a 'Golden Evaluation Set' Manually annotate a batch of 'question-standard answer' pairs, covering the core knowledge points of the knowledge base. Starting with 50~100 entries is enough, iterating along with the system.

2. Score Each Answer A lazy approach: directly let the LLM act as a judge for A/B scoring. Prepare a scoring Prompt and let the LLM compare the 'standard answer' with the 'model answer' to assign a score.

3. Log Q&A to Database Record query, retrievedChunks, finalAnswer, feedback (user likes/dislikes) for each Q&A session. After logging to the database, it's convenient to sample and review weekly to find counter-examples of 'recall failure'.

4. How to Tune When Metrics Are Poor — Locate First, Then Act

Phenomenon Problem Stage Tuning Direction
Retrieved chunk is incorrect Recall Stage Change chunking strategy / change embedding / adjust Top-K
Retrieved chunk is correct, but LLM didn't understand Generation Stage Modify prompt / upgrade model

Golden rule: Recall problems are more common than generation problems; check recall first. 90% of "AI answered wrong" cases are actually "didn't find the right material at all".

Token Consumption Control

Currently, token consumption is only controlled through Prompt input/output limits and incremental ingestion to reduce embedding model token usage. If higher-level models are used later, more refined control will definitely be needed, as they are genuinely expensive.

Automated Deployment

Recently, I've vibe-coded several Web Apps, and they basically all followed this process, which is quite convenient:

Note: You must first set secrets in GitHub and set environment variables in the relevant project directory on the server to avoid storing sensitive information in plain text.

Finally

Frankly speaking, this is just a simple RAG service, merely vector recall and model summary response (Naive RAG). In actual use, you'll find that when searching for extremely specific proper nouns, pure vector retrieval easily misses recalls.

I am also just starting with AI agent development, and my skills are still shallow. I hope passing experts will forgive me. There are actually advanced optimizations that can be done later, which I also plan to explore further in this column:

  1. Hybrid Search: Can overlay BM25 keyword search to solve the problem of missing professional terminology.
  2. Re-ranking: Introduce a specialized re-ranking model (like BGE-Reranker) to precisely filter out ineffective "similar nonsense."
  3. Agentic Flow: Integrate Function Calling to move from "Q&A" to "task execution," such as allowing it to retrieve content from other databases or search external resources online.

Embrace the AI era, get your hands dirty, see you in the next article.