跪拜 Guibai
← Back to the summary

Building a Production RAG Knowledge Base: DeepDOC Parsing, Hybrid Retrieval, and Pixel-Level Source Tracing

When building enterprise-grade large-model applications (such as mineral resource exploration and engineering technical specification Q&A), the core pain points of RAG (Retrieval-Augmented Generation) systems often lie in "inaccurate retrieval, incorrect answers, and inability to trace sources." A good knowledge base cannot stop at simple "chunking + vector retrieval"; it needs to balance high-precision multimodal parsing, hybrid recall, multi-level reranking, and precise source tracing.

This article deeply analyzes the full-chain implementation of this production-grade RAG system, following the order from macro pipeline to underlying core operators, and combines the real business scenarios of mineral exploration and geological survey materials to completely break down ingestion orchestration and performance bottlenecks, multi-format routing and in-depth PDF parsing, structured chunking and dual-mode index handoff, dual-channel hybrid retrieval and secondary scoring, and multimodal precise source tracing.

Core Architecture Overview

Untitled diagram-2026-10-25-052407.png

I. Full-Process Ingestion Pipeline and Engineering Time Consumption Analysis (Write Path)

After file upload, synchronous parsing is not performed; instead, decoupling is achieved through an asynchronous queue and data ingestion pipeline engine. The entire ingestion chain is uniformly driven by a Canvas DSL orchestration engine, decomposing tasks into 6 serial stages with independent states:

Untitled diagram-2026-10-25-052923.png

  1. Task queued / running (queue scheduling): The task is written to the NATS JetStream message queue and pulled for execution by the background Worker process of the ingestion service.

    Practical bottleneck insight: Under the default single-concurrency configuration (max_concurrent_workers = 1), only one task is executed at a time. In production, the phenomenon of "queueing for 7 minutes, actual execution only 3 minutes" often occurs. The primary way to optimize throughput is to reasonably increase Worker concurrency, rather than blindly optimizing single-point logic.

  2. [File] (read metadata): A lightweight pre-node that does not load the file binary content; it reads the file name, storage path, tenant ID, knowledge base ID, language, and other runtime metadata in seconds and broadcasts downstream.

  3. [Parser] (parse file content): The document parsing component pulls the file binary from object storage (MinIO) and routes parsing by suffix. PDFs go through the heavy DeepDOC visual engine, while office documents use a native lightweight parser. This is the stage with the highest computational load in actual execution.

  4. [Chunker] (smart chunking): The text chunker splits the parsed structured elements into chunks according to a strategy, supporting multiple configurable modes such as general, qa, table, and manual.

    Deep dive into the structured intelligent chunking engine:

    In the system architecture, [Chunker] is the "load-bearing wall" connecting front-end document parsing to back-end vector retrieval. If mechanical "fixed-length forced chunking" is used, it will inevitably cut a complete geological structure description, a complex drill hole assay table, or a key prospecting conclusion in the middle. To this end, the system has built-in multi-strategy configurable chunking modes:

    • 1. General (general/standard chunking):

      • Core logic: Uses a "sliding window + natural semantic boundary awareness" strategy, with chunk_token_size (default 512 tokens) as the soft upper limit, preferably breaking at natural delimiters (such as 。, ？, ！, ;, \n), supporting overlap ratio (Overlap) configuration.

      • Geological practice: Suitable for long continuous texts like "Regional Geological Background", ensuring smooth splitting of long expositions into semantically complete chunks.

    • 2. QA (question-answer pair chunking):

      • Core logic: Designed for Q&A pairs, specification clauses, or FAQ structures, identifying the alignment structure of "question (Q) and answer (A)" and forcibly binding them within the same chunk.

      • Geological practice: Suitable for "Question and Answer on the Classification Standard for Solid Mineral Resources/Reserves". When a chief engineer asks a question, this chunk can reach precisely with zero latency.

    • 3. Table (table-specific chunking):

      • Core logic: Maintains the integrity of the table structure and absolutely does not cut horizontally from the middle; if the table has too many rows, the system uses a "Row-Header Inheritance" mechanism that automatically attaches the complex header as a context prefix to each split row chunk.

      • Geological practice: Handles hundreds of rows of drill hole logs or assay tables, ensuring that a single row of assay data can still restore its physical meaning through the header during chunking and retrieval.

    • 4. Manual (book-level hierarchical chunking):

      • Core logic: Based on the heading hierarchy (H1, H2, H3) output by DLA, chunks are made by chapter topology, with "complete sub-chapters/technical clauses" as boundaries, and each chunk header automatically injects a macro breadcrumb navigation path (such as [Chapter 1 Mining Area Geological Characteristics] > [Section 3 Ore-Controlling Structures]).

      • Geological practice: Handles hundreds of thousands of characters of "DZ/T 0214-2002 Exploration Specification", ensuring the large model not only knows the content of the clause but also knows the chapter it belongs to.

  5. [Extractor] (LLM extraction enhancement): An optional information extraction component. When enabled, it concurrently calls the large model for each chunk (the concurrency cap is controlled to 10 by default) to generate high-dimensional semantic enhancement information, addressing the pain point of "professional documents being obscure and having a semantic gap with user colloquial questions":

    • Auto-Keywords: Extracts core entities and domain terms from the chunk (for example, from core logging descriptions, extracts high-weight words such as 斑岩型铜矿, 钾硅酸盐化, 黄铜矿化富集带, 钻孔ZK402), and assigns higher term weights when writing to the index.
    • Auto-Questions / reverse HyDE: From the questioner's perspective, predicts what questions this content can answer (for example, for ore deposit genesis, automatically generates: "What is the ore-controlling structure of the gold ore body in the eastern part of the research area?"). These generated questions participate in tokenization and vectorization along with the original text, transforming "question searching statement" into extremely high-match "question searching question".
    • Summary & Tags: Condenses long paragraphs into core conclusion summaries, or automatically applies business tags (such as 地球化学勘查, 水文地质条件) according to a preset system.
    • Engineering cost: If this feature is not enabled, this node simply skips. Once enabled, each chunk requires an LLM request, becoming the second-largest time consumer after the Parser.

  6. [Tokenizer] (tokenization, vectorization, and disk write): The tokenization and vectorization component generates full-text retrieval token strings and dense vectors, finally writing to the underlying pluggable retrieval engine layer.

    Storage engine comparison (Elasticsearch vs. Infinity):

    The system does not bind to a single database at the bottom layer; instead, it supports two differentiated engines through a unified abstraction interface:

    • Elasticsearch (ES): The traditional distributed full-text search hegemon based on Java/Lucene, with many existing enterprise clusters and mature operations ecosystem; however, its vector retrieval capability was added in later evolution, and memory overhead is relatively high when handling large-scale high-dimensional vectors and full-text hybrid computation.
    • Infinity: A C++ native AI database built specifically for large models and RAG. It natively supports high-speed fused queries of "dense vectors + sparse vectors + full-text search" at the bottom layer, with extremely high single-machine throughput and far lower memory usage than ES. Enterprises can seamlessly switch based on their infrastructure.

II. Core Difficulties: Multi-Format Routing and In-Depth PDF Parsing

In the [Parser] stage, the system faces a variety of document formats within the enterprise. To balance parsing speed and restoration accuracy, the parsing scheduler adopts a "routing by suffix, separating heavy and light" design philosophy.

1. Multi-Document Format Parsing Strategy Overview

Streaming office documents with built-in structure trees directly use native syntax tree (AST/DOM) extraction, while unstructured or layout-fixed PDF documents are handed over to the DeepDOC visual pipeline:

Document Type Included Suffixes Core Parsing Method and Processing Strategy Computational Cost
Layout-fixed documents .pdf Uses the DeepDOC two-stage visual pipeline (OCR + DLA layout analysis + TSR table recognition), extracting text, tables, images, and precise page coordinates Extremely heavy (CPU-intensive)
Word processing documents .docx, .doc Based on the underlying XML structure tree (AST), directly extracts paragraph text, outline hierarchy, embedded tables (converted to HTML/structured blocks), and embedded images Light (milliseconds/seconds)
Spreadsheets .xlsx, .xls, .csv Natively reads by worksheet (Sheet) and row/column structure, supports associating headers with data rows, or converts to HTML <table> / key-value text blocks, no visual TSR required Light
Presentations .pptx, .ppt Iterates by slide (Slide), parses text boxes, shape text, tables, and notes areas within the page, concatenating in geometric order from top to bottom, left to right Light
Web and markup languages .html, .md Parses DOM tree or Markdown AST, strips noise tags like <script>/<style>, retains heading hierarchy (H1~H6), lists, and table structures Very light
Plain text .txt Automatically detects character encoding (UTF-8 / GBK, etc.), reads directly by natural newlines and delimiters Very light

Streaming office documents (such as .xlsx drill hole assay data tables, .docx field geological reconnaissance summaries) naturally maintain the logical tree of paragraphs and tables, allowing the lightweight parser to achieve 100% accurate restoration. What truly makes most RAG systems "fail" are PDF geological survey reports with free-form layouts, lack of unified logical trees, and even historical scanned documents. To this end, the system has fully ported and optimized the DeepDOC two-stage deep parsing engine (aligned with the Python original) in Go.

2. PDF Stage One: Page-by-Page Independent Parallel Pipeline

After a PDF enters the parser, it is first scheduled by a page concurrency worker pool, with each page independently and sequentially passing through 1 pre-detection step and 5 core steps. Normal PDFs and scanned documents go through the same pipeline, with the only fork point in step two:

  1. Pre-step: Extract embedded character layer - the premise for forking. Calls the underlying PDF rendering library (pdfium) to extract embedded characters and their precise coordinates. Normal PDFs return a large number of per-character coordinate objects (at this point there is no concept of "line"); scanned documents return empty. Also calculates median character height/width (for judging font size during paragraph merging), English page determination, and scanned noise page determination - identifying PDFs with a character layer that is actually garbage/noise, forcing them to be treated as scanned documents.

  2. Step One: Render 216 DPI bitmap - uniform input for visual models. Regardless of whether there is a text layer, the rendering engine converts the page to a 216 DPI bitmap. This is because subsequent line box detection (Det), layout analysis (DLA), and table recognition (TSR) are all visual models that only look at pixels, not the text layer.

  3. Step Two: Obtain "line-level text boxes" - the only fork point. The system dynamically routes between two processing paths based on whether the page has a clean embedded character layer (has characters, is not scanned noise, and is not a garbled page) (see comparison table below).

  4. Step Three: DLA layout analysis - assign semantic roles. Runs a layout detection model on the entire page bitmap, outputting 10 types of layout region boxes, and maps the layout role labels back onto the text boxes produced in step two (illustration regions also get a synthetic box placeholder).

  5. Step Four: TSR table structure recognition - decompose complex grids. Triggered only when step three DLA detects a table region: crops the table image from the page image -> runs the TSR model to recognize rows, columns, headers, and merged cells spanning rows/columns -> assembles cells into a grid and fills in the cells using the text from the step two line boxes (for empty cells that cannot be filled, crops the image again and runs OCR) -> annotates each text box with row/column position for later assembly.

  6. Step Five: Zero-output zoom retry (Zoom Retry). If after the first 4 steps an entire page produces no boxes (usually due to low original resolution or very small text), it triggers a zoomed re-render and re-runs OCR + DLA.

3. Three Core Difficulties and Architectural Design Breakdown

In the page-by-page pipeline, there are three key designs that determine system performance, stability, and accuracy:

Difficulty A: Cost and Path Forking for Normal PDF vs. Scanned Documents (Breakdown of Step 2)

Processing Dimension Normal PDF (reuse boxed characters) Scanned / garbled pages (full visual detection and recognition)
Core Idea The text layer has characters but no "line" structure; the visual model has "line boxes" but reads characters slowly - use detection boxes to enclose embedded characters Pure image without text layer, must go through the full "detection + recognition" visual pipeline
Execution Steps 1. Run OCR Detect on the bitmap to get line-level quadrilateral boxes 2. Merge embedded characters into corresponding line boxes by coordinate overlap 3. Directly concatenate characters within boxes; only for empty boxes or lines where special font encoding fails (such as geological era rare characters, ore chemical formulas) crop the image and run OCR Recognize to fill gaps 1. Run OCR Detect on the bitmap to get line-level boxes 2. For each box, perform perspective correction and crop (for vertical blocks with aspect ratio > 1.5, generate additional clockwise/counterclockwise 90° candidate images, common in geological column sidebars) 3. Sort by aspect ratio, batch every 16 images and run OCR Recognize in full, output the direction with the highest confidence
Single-page inference cost 1 render + 1 Det + 1 DLA (+1 TSR per table), Rec calls approach zero, extremely fast On top of normal PDF, run N additional Rec (N = number of rows on the page), with pages of hundreds of rows requiring dozens of batches of inference, which is the root cause of slow scanned documents

Architecture distinction: Built-in DeepDoc dedicated models vs. UI VLM enhancement plugin

  • Four built-in DeepDoc models (ONNX): Infrastructure packaged with the product (Det, OCR Rec, DLA, TSR), inferenced locally in-process on CPU via ONNX Runtime, no configuration required, zero token cost. They are only responsible for "locating boxes, recognizing characters, labeling layout, decomposing table structure", and do not understand what is depicted in the image (they only know this is figure, not whether it is a fault cross-section or a mining area traffic map).

  • External VLM enhancement plugin (e.g., GPT-4o-mini): Tenant-level remote API configuration. After parsing, the illustration/table crops marked by DLA are sent to the VLM to describe the image content in natural language (e.g., output: "Figure 3-2 shows the cross-section relationship of the NE-trending F1 reverse fault cutting through granite porphyry, with the main ore body in the fracture zone at the footwall"), converting to searchable chunks. Without VLM, the main flow runs normally, only lacking image semantic retrieval capability.

Difficulty B: DLA Layout Analysis's 10 Semantic Labels (Breakdown of Step 3)

The class indices output by the layout analysis model strictly map to the 10 semantic roles defined in the layout label definition table:

Number Label Name Semantic Meaning Downstream Pipeline Processing Strategy
0 title Title (chapter name, document main title) Writes back to text box layout attributes, identifies hierarchy and title paragraphs during paragraph assembly
1 text Body (ordinary paragraph text) Default main content, participates in regular vertical paragraph merging
2 reference References (end-of-document citation entries) Marked as reference area, isolated from main body
3 figure Illustration/image area (geological maps, column charts, etc.) Merged into independent figure blocks, synthetic boxes added and images cropped and saved
4 figure caption Figure caption (e.g., "Figure 1: Mining area geological sketch") Not an independent paragraph, automatically attached to adjacent figure during assembly
5 table Table area (e.g., exploration project list) Triggers step 4 TSR, crops image to recognize row/column grid, headers, and merged cells
6 table caption Table caption (e.g., "Table 1: Ore grade analysis table") Not an independent paragraph, automatically merged into corresponding table during assembly
7 table caption Table caption (same as 6, protocol reserved duplicate) Same as number 6
8 equation Formula area (e.g., reserve estimation mathematical formulas) Marked as independent formula block, avoiding being merged into ordinary body paragraphs by vertical merging logic
9 figure caption Figure caption (same as 4, protocol reserved duplicate) Same as number 4

Difficulty C: Three Layers of Hard Protection Against Memory Explosion (Breakdown of Step 5)

In step 5's Zoom Retry, bitmap memory grows quadratically with the zoom factor. For pathological scanned pages, directly throwing them into the underlying renderer would take minutes to allocate local memory or even trigger OOM (out-of-memory crash).

Real calculation example of geological large image OOM:

In mineral exploration archive, reports often end with A0 super-large-format "1:10000 Regional Geological Topographic Map" or "Exploration Line Combined Profile" (physical size approximately $46.8 \times 33.1$ inches).

  1. First rendering: At the base 216 DPI, pixels are approximately $10110 \times 7150 \approx 7200\text{ 万 pixel}$ (actually near the budget cap due to underlying clamping).

  2. Disastrous unprotected zoom: If the first round "zero output" triggers a 3x zoom retry (648 DPI) because the map surface has large whitespace or the legend text is too small, width and height each multiply by 3, total pixels surge 9 times to $6.5\text{ 亿 pixel}$!

  3. Memory crash: Just this one uncompressed RGBA bitmap requires $6.5\text{ 亿} \times 4\text{ Bytes} \approx \mathbf{2.6\text{ GiB}} $ of continuous native memory; adding temporary coexistence of old and new bitmaps and intermediate feature tensors during ONNX visual model inference, single-page instantaneous memory consumption breaks through 4~5 GiB. If at that moment the worker pool happens to have 4 concurrent workers processing similar fold-out large images, the instantaneous near 20 GiB memory flood will instantly break the container memory limit and cause the process to be OOM killed.

The system builds a triple defense:

  1. Narrow trigger and multiplier cap: Only first-round "zero output" pages trigger retry; the retry multiplier is hard-capped at 9.0x.

  2. Pre-render pixel budget pre-check: Before entering the underlying renderer to allocate memory, first estimate the new bitmap size by multiplying the first-round width/height by the square of the zoom factor. Once it exceeds the 48M pixel (~48 million pixel) budget, the retry is abandoned and the first-round result is retained, directly blocking the dangerous memory allocation of the above geological fold-out large images.

  3. Internal fallback clamp in the renderer: The underlying PDF rendering engine has its own ultimate defense. If the computed pixel exceeds the limit, it automatically proportionally reduces the resolution scale (scale), ensuring that any single rendered bitmap can never exceed 48M pixels (~192 MiB RGBA). The memory peak of the entire parsing pool is strictly converged to a bounded safe range of concurrent worker count × ~200 MiB.

4. PDF Stage Two: Document-Level Global Assembly and Practical Full-Process Walkthrough

After all page text boxes, tables, and DLA regions are collected by page number, they enter the document-level layout assembly stage for cross-page global organization:

  1. Column detection: First identify double/multi-column layouts and label column numbers (must be executed before deduplication to prevent similar short lines across columns from being mistakenly deleted).

  2. Global noise cleaning: Remove overlapping boxes and background watermark text (e.g., italic full-page "Internal Geological Data · Strictly Prohibited for External Distribution"); automatically delete headers and footers through cross-page statistics (such as the "XX Province XX County Gold Copper Ore Detailed Survey Report" appearing at the top of more than half the pages); combine PDF bookmarks and morphological features to remove table of contents (TOC) pages entirely.

  3. Paragraph restoration: Execute same-line fragment merging -> arrange reading order by column -> reconnect English cross-line hyphens -> merge same-role text vertically into natural paragraphs.

  4. Table finishing and image-text alignment: Perform cross-page table merging, extract scattered table boxes from the body text flow, and replace them with structured HTML table objects; finally convert the text box flow into final structured paragraphs (Sections) and attach figure captions and table captions precisely to the corresponding image/table objects.

Full-process walkthrough of prospecting report parsing (using "Detailed Survey Report of a Porphyry Copper-Molybdenum Deposit in a County, Qinghai Province.pdf" as an example):

Assume this is a PDF report mixing 1990s historical scanned pages and modern electronic typesetting, where pages 42~43 contain double-column body text, a "Figure 4-1 Plane Map of IP Anomaly in Mining Area", a cross-page "Table 4-2 Drill Hole Ore Intersection and Chemical Analysis Results Table", and a mathematical formula for estimating copper metal quantity using industrial indicators:

  1. Page-by-page routing: Page 42 is a modern electronic page; the system directly uses detection boxes to enclose embedded characters, zero OCR recognition, seconds to output text. Only the ore chemical formula $\text{CuFeS}_2$ due to missing special font encoding is automatically cropped and run through OCR once; page 43 is a hand-stamped scanned page from an old geological team, automatically switching to full OCR path, and the vertical "rock name" column in the table is automatically rotated 90° for accurate recognition.

  2. Role assignment: DLA marks the double-column text as text, marks the reserve calculation formula $Q = V \cdot d \cdot \bar{C}$ as equation independently protected; marks the IP anomaly map and its below description as figure and figure caption respectively.

  3. Cross-page table stitching: TSR recognizes the half table at the bottom of page 42 and the continuation table at the top of page 43 (including complex two-level merged headers such as "ore body number", "drill hole number", "true thickness (m)", "average grade Cu(%) / Mo(%)"). During global assembly, the system automatically removes the headers/footers interleaved between the two pages, seamlessly splices the two tables into a complete HTML <table>, and attaches the table caption precisely above the table header.

III. Structured Chunking and Dual-Mode Index Construction (Data Handoff)

After the parsing stage, whether PDF or Office documents are uniformly converted into structured text paragraphs (with layout type and page coordinates), HTML table objects, and image resources. Next, the data handoff goes to the chunker and tokenizer/vectorizer components:

  1. Structured intelligent chunking (Chunker) :

    • Regular text splitting: Taking the general chunker as an example, the system merges or splits text segments into semantically complete chunks based on natural delimiters (\n, 。, ！, ？, ;, etc.) and token thresholds (default 512 tokens), while supporting overlap ratio (Overlap) and sub-chunk delimiter configuration.

    • Independent protection of images/tables and coordinate inheritance: HTML tables and image regions with VLM descriptions are not mixed with regular body text; they are kept as independent chunks (or with a context window), preventing the "Drill Hole Ore Intersection Results Table" from being cut in the middle. At the same time, when multiple PDF text boxes merge into one chunk, their page numbers and rectangular coordinates (Bounding Box) are completely inherited into the chunk metadata as an array.

  2. Dual-mode index and feature persistence (Tokenizer) : Each chunk (optionally stacked with LLM-extracted keywords/questions) produces two core data paths in parallel:

    • Full-text index side: Calls the underlying C++ tokenizer binding to generate fine-grained token strings (content_ltks / content_sm_ltks) and title tokens (title_tks), solving compound matching problems for geological proper nouns.

      Example of two-granularity tokenization for geological terms:

      Suppose the chunk original text is: "The NE-trending ductile shear zone controlled the spatial occurrence of altered rock-type gold ore bodies, and drill hole ZK305 encountered pyritic sericite-quartz altered cataclasite at a depth of 215m."

      • Coarse-grained tokenization (content_ltks): Produces 北东向 / 韧性剪切带 / 控制 / 蚀变岩型金矿体 / 空间产出 / 钻孔 / zk305 / 孔深 / 215m / 黄铁绢英岩化 / 碎裂岩. Ensures high-precision phrase matching when users search for the professional long term "蚀变岩型金矿体".

      • Fine-grained secondary splitting (content_sm_ltks): Further decomposes compound long terms into 韧性 / 剪切 / 剪切带 / 蚀变 / 蚀变岩 / 金矿 / 矿体 / 黄铁 / 绢英岩. This way, even if a geologist only inputs abbreviations like "绢英岩金矿" or "剪切带控矿", high recall can still be achieved through fine-grained terms, completely eliminating the problem of long-tail compound terms not being searchable.

    • Vector index side: Calls the Embedding model in batches according to the model's maximum batch limit to generate dense vectors. The system has a built-in per-chunk vector cache mechanism. Combined with Canvas DSL node state management, when a task is interrupted due to network jitter and rerun, the already computed vectors can be reused, enabling low-cost resume from breakpoint.

    • Final write: Token strings, dense vectors, page coordinate metadata, along with document-level graph features (PageRank features) used to boost the ranking weight of core authoritative documents (such as final reviewed and archived reserve reports), are all persisted to the Elasticsearch or Infinity retrieval engine.

IV. Read Path: Dual-Channel Hybrid Retrieval and Secondary Scoring Service

When a user asks a question, the system enters the core hybrid retrieval service, which uses a two-stage design of "engine-level coarse recall + service-level precise scoring" to balance high recall and high accuracy:

  1. In-engine dual-channel hybrid recall (Hybrid Retrieval) :

    • Sparse side: The query constructor tokenizes the user question to generate a full-text Match retrieval expression.

    • Dense side: The Embedding model converts the question into a query vector to construct a vector kNN retrieval (default topK = 1024).

    • The search engine executes both paths in a single query request and pulls the preliminary merged candidate set through the engine's built-in Fusion expression.

  2. Service-layer secondary scoring (Rescore) and Rerank fine-ranking : After obtaining candidate chunks, the retrieval service layer performs explicit secondary weighted scoring based on precise term matching and vector cosine similarity:

    $\text{sim} = (1 - w) \cdot \text{termSimilarity} + w \cdot \text{vectorSimilarity}$

    where $w$ is the vector similarity weight (default 0.3, meaning default 0.7 weight favors keyword exact matching, 0.3 weight allows semantic generalization via vector).

    • Rerank coverage: If the knowledge base is configured with an external Rerank cross-encoder model, the system performs deep semantic cross-ranking on the top N candidates (default 64), with the Rerank score dominating the final ranking; if not configured, the above hybrid weighted score is used directly.

  3. Threshold filtering and intelligent fault-tolerance fallback :

    • Threshold truncation: Filters low-score noise chunks by similarity threshold (default 0.2), sorts and paginates to return Top Chunks and document-level aggregate statistics.

    • Degraded retry fallback: If the first retrieval encounters "zero hits", the system automatically relaxes conditions (lowering the minimum match ratio of full-text retrieval and lowering the vector threshold) for a second retry; it also provides a pure vector (VectorOnly) retrieval mode for purely descriptive queries that skips keyword restrictions.

Full-chain practical example of prospecting retrieval:

Suppose a chief engineer asks in the dialog: "What is the grade of associated gold in the copper-rich ore body encountered at depth by drill hole ZK305? What alteration occurred in the surrounding rock?"

  1. Why does single-channel retrieval fail?

    • If using pure vector retrieval only: ZK305 is extremely close in dense vector space to ZK301 and ZK308, making the vector model prone to "confusing identities", ranking the copper-rich ore description of the neighboring ZK308 first, causing the large model to answer with wrong data.

    • If using pure keyword retrieval only: The report original text says "copper-bearing pyrite ore body" and "propylitization, silicification strong", and does not directly contain the terms "copper-rich ore body" and "wall rock alteration" from the question, making pure keyword matching prone to missed recall.

  2. How does dual-channel hybrid + secondary scoring break the deadlock?

    • The sparse side locks onto the precise engineering number zk305, associated, au (gold), grade through tokenization; the dense side uses semantic generalization to capture that "propylitization/silicification" is "wall rock alteration" and "Cu 2.85%" is "copper-rich ore body".

    • During service-layer secondary scoring, because the default term weight is 0.7 ($1 - 0.3$), the chunk containing the precise number ZK305 receives near-perfect termSimilarity, instantly pushing interfering drill holes like ZK308 behind.

  3. Rerank final strike: After the top 64 candidate chunks are sent to the Rerank model, the cross-encoder deeply compares the question and candidate chunks, accurately pushing the Table 4-2 on page 42 (Drill Hole Ore Intersection Chemical Analysis Results Table, including ZK305 Au grade 0.85 g/t) and the page 58 "ZK305 Drill Hole Geological Log Summary" (recording strong silicification and propylitization at depths 210~235m) to Top 1 and Top 2 (comprehensive scores above 0.85, far exceeding the 0.2 threshold), with the remaining low-score chunks cleanly cut off.

V. Generation Landing: Context Injection and Multimodal Precise Source Tracing

Each Chunk returned by the retrieval service not only contains text content but also fully carries the three-dimensional score components of comprehensive similarity, term similarity, and vector similarity, as well as positional metadata transparently passed through from the parsing stage.

The upper-layer Chat/Agent runtime module assembles these high-value Chunks into the Prompt context, driving the large model to generate answers with citation corner markers. On the front-end source tracing interaction, the system implements tiered precise back-linking for different document types:

Through this complete closed loop of "multi-format deep parsing -> structured dual-mode ingestion -> hybrid recall and reranking -> pixel-level coordinate back-linking", the system truly achieves the production-grade requirements of an enterprise RAG knowledge base: search accurately, answer correctly, and verify with evidence.