Citi, EY, and Nvidia's Document Ingestion System Report: 92% of Pages Don't Need OCR

💡 One-sentence summary: This is an MLSys industrial-track report with no new algorithms, but anyone who has worked on enterprise RAG will nod hard: three authors from Citi, EY, and Nvidia turn the dirty work of "how documents become vectors a model can eat" into a systematic four-phase pipeline, with evidence attached at every step. The most useful number: after page-level routing, 92% of pages in enterprise corpora go through zero-model-cost native extraction, and only 8.1% need an OCR model — if you are still OCR-ing everything without thinking, that is money you are right to save.

Introduction: Nobody Owns the "Perception Layer" of the GenAI Era

In the traditional machine learning era, data preparation was narrow: the task was fixed, the schema was fixed, and data cleaning was aimed at the objective function. Generative AI freed the model — one foundation model serves countless downstream tasks — at the cost of pushing all the data-side complexity to the entry: enterprise knowledge lives in human-facing formats, PDF, PPT, Excel, scans, forms, mixed layouts, and a single file can contain text, charts, geometric layout, and structural information at the same time.

If this entry pipeline is broken, no matter how strong the downstream retriever or reranker is, it cannot be saved — this is a lesson repeatedly verified in our earlier pieces (EVICALC, TRACE, FoveDoc). But this paper asks an even earlier question: how should this entry pipeline itself be built, tested, and managed?

Three authors (Citi's Abbas Raza Ali, EY's Muhammad Ajmal Siddiqui, and Nvidia's Moona Zahid) present a four-phase production system, submitted to the MLSys 2027 industrial track. The paper's self-positioning is honest: it proposes no new algorithms; its contribution is "routing, scoring, and audit scaffolding around existing parsers." The whole paper also has an endearing self-discipline: every claim is tagged with an evidence level: [E] established research, [D] the system's own measurements, [P] proposed here, [H] hypothesis. An enterprise system report that does this is itself exemplary.

What's the Approach? Three Principles + Four Phases

First, three design iron rules to keep in mind:

1. Structure before semantics: anything that can be solved with the document's own structure (font hierarchy, table lines, headings, pagination) should not use a learned model — deterministic, cheap, debuggable.
2. Don't change what you measure: the evaluator is read-only and reuses the exact models and configurations from production — evaluation must test the production system.
3. Budget annotation: full ground truth is unrealistic; use a small, modality-balanced, traceable sampled subset to stand in for population confidence.

Phase 1, extraction: make OCR the exception. Each page gets an image-area ratio; only pages above 0.8 are rasterized and sent to a vision model. Everything else goes through PyMuPDF native structure extraction: font size calibrates heading hierarchy; blocks that reappear across pages after number normalization are removed as headers/footers; lined tables use line detection; wireless tables use word-alignment clustering (with a guard that says "this looks like multi-column prose" to prevent collateral damage). On the Orion corpus's 10,507 pages, only 8.1% of pages go through OCR. One detail shows the craft: the curation stage detects "scan pages" with two signals — a text layer under 40 characters and a high image share — to avoid misclassifying "a full page of text with one large photo."

Phase 2, curation and scoring: the people handing out exams to extractors have to be reliable. Two components. Scarcity-first sampling: when a corpus is 90% plain text, random sampling starves scan-page and chart-page samples — so it does greedy coverage by modality quota (pick the rarest modality first, and when picking, prefer pages that can also satisfy other quotas), with a six-table audit workbook to keep every choice reproducible. A reference scorer: after normalization, it computes CER/WER/BLEU/length ratio, uses TEDS tree edit distance for tables, and finally combines them into one score — the key move is multiplying by coverage: a model that "skips all hard documents and only solves easy ones" cannot score high. This anti-cheating design is worth copying by everyone who runs model leaderboards.

Phase 3, chunking: small cards, big folders. The granularity for retrieval and the granularity for reading should not be the same: child cards are fixed at 320 tokens (max 512) and go into the vector index; parent paragraphs (≤3500 tokens) live in a KV table. At query time, a card is hit and the folder is fetched in O(1) to feed the model. Among the four splitting rules, the most worth learning is "tables and figures are atomic" — never cut them. Plus a 48-token overlap seed, so a cross-boundary answer appears complete on at least one card. Each child card's embedding text is prefixed with a structural breadcrumb, "document > title > section > subsection (p.12)" — a deterministic version of contextual retrieval, at zero LLM cost.

Phase 4, read-only retrieval evaluation. Use Mistral-Medium to generate "answerable within the page" questions from each page (external knowledge strictly forbidden), retrieve with the same fingerprint embedding model used in production, and the hit criterion is similarity threshold met and page distance no more than ±1. The evaluator is read-only — it can never pollute the online index.

How Well Does It Actually Work?

• Extraction (coverage-penalized composite score): Gemini 3.1 Pro leads with 97.37 (CER 0.13%); Mistral OCR v3 scores 95.98 (CER 1.39%, table TEDS still above 0.97); two compact Docling versions trail by 4-7 points, with the gap mainly in table structure. As a reference, the authors' earlier EDD-based approach had only 85% word accuracy on image tables — the production backend's error rate has dropped by one to two orders of magnitude, which shows how fast this field is moving.
• Latency: full extraction of 180 documents takes 45-57 minutes, but about 34 minutes of that is fixed preprocessing cost shared by all backends. After 92% of pages go through native routing, model inference is no longer the bottleneck; the next optimization target is preprocessing.
• Retrieval: on 25,050 synthetic questions, hit@1 is 68.6%, hit@10 is 92.8%, MRR is 0.77, median rank is 1. Errors concentrate in the long tail rather than being evenly distributed.

What Can You Use This For?

• Reference architecture for enterprise RAG ingestion pipelines: four-phase structure with evidence artifacts at each phase; you can almost plan development directly from it.
• Save OCR budget: page-level routing plus dual-signal scan-page detection can save over 90% of vision-model calls on most enterprise corpora.
• Anti-cheating for evaluation leaderboards: composite score times coverage — specifically for models that cherry-pick.
• Chunking strategy: atomic tables + parent-child decoupling + breadcrumb prefix is a deterministic solution with zero external dependencies (no LLM).
• Evaluation discipline: read-only evaluator + production fingerprint reuse prevents chronic problems like evaluation/production drift.

Reality Check

• All numbers are self-tests of the system (marked [D] in the paper): internal corpus, no public code or datasets; reproduction can only rely on the description. "Auditable" is an architectural property, not external validation.
• The retrieval evaluation questions are LLM-synthesized, and generation quality depends on the generator itself. The ±1 page distance in the hit criterion is also a design choice.
• Only tested up to the retrieval layer: hit@k and MRR are not end-to-end answer quality; generation-stage performance is unknown.
• The sample size for extraction ground truth is not disclosed. Scarcity-first sampling is cleverly designed, but confidence that a small sample represents the population still depends on annotation quality.
• The 0.81s single-query latency is mostly contributed by remote embedding calls. For latency-sensitive scenarios, that number is not pretty.

A Rational View

This paper's value is not in any single point technology — routing, sampling, chunking, and evaluation, taken apart, are engineering of known components. It lies in elevating "ingestion" from glue code nobody wants to claim into a first-class stage with interfaces, metrics, and an audit trail. Among the three iron rules, "don't change what you measure" is especially worth a self-check by every data team: does your evaluation environment and production environment use the same tokenizer, the same embedding model, the same set of regexes? If not, those pretty offline metrics were never describing your system from the start.

Looking at this series together is more interesting: earlier papers proved that "if the entry representation is wrong, no downstream god can save you"; this one gives the engineering answer for the entry pipeline. For a team doing RAG on enterprise documents, this may be the one most like a "construction drawing" among the fifteen — it is not cool, but it aligns the baseboards.

(Data and conclusions come from the original paper; the interpretation is the author's view. Comments welcome.)