CPU-Only Invoice OCR: 4 Post-Processing Traps Between Text Lines and Structured Data

Invoice Recognition on CPU: 4 Hard Problems from OCR Text Lines to Structured Line Items

A locally-run offline VAT invoice recognition service that requires no cloud API or GPU, and outputs structured invoice fields and product line items.

Why this was built

I had a 73-page invoice PDF and wanted to read out the line-item table—product name, specification, quantity, unit price, amount, tax rate, tax amount—without a single column going wrong.

This sounds like an OCR problem, but it is not. OCR only turns the image into "a pile of text with coordinates" (shown here as a four-point box):

[{"text": "*生物制品*胎牛血清", "box": [[53,412],[186,412],[186,441],[53,441]]},
 {"text": "500ml",             "box": [[201,415],[248,415],[248,438],[201,438]]},
 {"text": "瓶",                "box": [[265,415],[281,415],[281,438],[265,438]]},
 ...]

Going from that pile of coordinates back to "line item 1: fetal bovine serum, spec 500ml, unit bottle, quantity 2"—that middle part is where the real time goes. It consumed most of the project's effort.

Why not just use an off-the-shelf solution

The choices that were available each have their own pain points:

	Cloud OCR API	General open-source OCR (PaddleOCR etc.)
Recognition accuracy	High	Medium
Where invoice data lives	Uploaded to third-party cloud	Local
Cost	Per-call billing	Free
Structured invoice fields	Built-in	Almost none, you have to write them from scratch
Deployment barrier	Low	High, often needs GPU

PaddleOCR can recognize the four characters for "fetal bovine serum", but it does not know that is a product name, which line it belongs to, or which specification it matches. All of that relies on post-processing.

So this project fills that missing middle: local CPU inference that directly outputs structured invoice fields and line items. The request and response formats align with Tencent Cloud's VatInvoiceOCR, so any code that already used the cloud API can switch over by just changing the address.

The code shown here is an excerpt (logging and edge-case handling omitted). The full implementation lives in the repository's app/extract/ directory.

Overall flow

Image / PDF
  ↓  For PDFs, render at a scale derived from the 2000px long edge (a fixed DPI would make small receipts too large and large ones too small)
RapidOCR (PP-OCRv6 + ONNX Runtime, CPU inference)
  ↓  Text lines + four-point coordinates
Layout reconstruction: cluster lines by y, sort by x within a line
  ↓  List of rows
Find the table header → derive each column's boundaries from the header tokens' x centers
  ↓  Column intervals
Assign line items to columns → merge wrapped lines → validate numeric values → align the total row
  ↓
Extract invoice header fields + cross-checks (amount reconciliation, tax reconciliation)

Each of the last four steps has a trap. They are described one by one below.

Hard problem 1: Wrapped text goes back to which column?

Long product names wrap. On the invoice it looks like this:

*生物制品*胎牛血清    500ml      瓶      2     294.247...
胎牛血清（澳洲来源）

The second line has only text, no numbers. The most intuitive approach is to append the whole line to the product name, which produces "胎牛血清（澳洲来源）500ml 瓶"—spec and unit have been stuffed into the name.

The real situation is worse: OCR splits the wrapped line by columns, so the continuation tokens land separately in the name column, spec column, and unit column. For example, the spec 10只/包，500只/箱 wraps to two lines, and the second line's 只/箱 lands in the spec column.

So you cannot append the whole line. You have to decide each chunk's ownership from left to right by x coordinate:

def _merge_continuation(pending, ordered):
    """Merge a line that has only text and no numbers into the previous line item."""
    for _, column, text in sorted(ordered, key=lambda pair: pair[0]):
        value = clean_value(text)
        if not value:
            continue

        if column == "Spec" and _spec_continues(pending.get("Spec", ""), value):
            pending["Spec"] = clean_value(pending["Spec"] + value)
            continue

        if column == "Unit":
            if not pending.get("Unit"):
                pending["Unit"] = value
            elif _spec_continues(pending.get("Spec", ""), value):
                # "箱" was cut into the unit column, but it is the tail of spec `50只/`
                pending["Spec"] = clean_value(pending["Spec"] + value)
            elif value != pending["Unit"]:
                # Non-unit text in the unit column (like `无`) is a continuation of the product name
                pending["Name"] = clean_value(pending["Name"] + value)
            continue

        pending["Name"] = clean_value(pending["Name"] + value)

The core is _spec_continues: whether this piece of text is the second half of a spec, or a continuation of the product name.

def _spec_continues(spec: str, text: str) -> bool:
    if not spec or not text or len(text) > 8:
        return False
    if spec.endswith(("/", "／", "，", ",", "、")):
        return True          # Spec clearly unfinished, e.g. `1只/包，50只/`
    return bool(re.search(r"[/／]", text))

Both conditions are plain but effective: the previous chunk ends with / or a comma (meaning it was not finished), or the current chunk contains / (only specs use slashes).

After this code went live, a product name like 袋装无酶 no longer got stuffed into the spec column when it wrapped.

Hard problem 2: Quantity and unit price get glued into one number by OCR

The invoice shows "quantity 1" and "unit price 397.34..." as two separate boxes, but when they sit close together, OCR outputs a single 1397.34....

The hard part is: how do you know the 1 is the quantity and the 397.34 is the unit price, rather than quantity 13 and unit price 97.34?

Use a constraint: unit price × quantity ≈ amount. The amount is a separate column and usually does not go wrong, so it can be used to reverse the split:

def _reconcile_qty_price(item):
    amount = to_float(item.get("AmountWithoutTax", ""))
    raw = nfkc(item.get("UnitPrice", "")).replace(" ", "")
    ...
    price, qty = to_float(raw), to_float(item.get("Quantity", ""))
    if abs(price * (qty if qty else 1) - amount) <= 0.02:
        return  # Already self-consistent, leave alone

    for k in (1, 2, 3):                      # Try at most 3 digits as the prefix
        head, rest = raw[:k], raw[k:]
        if not rest or not (rest[0].isdigit() or rest[0] == "."):
            continue
        q, p = float(head), float(rest)
        if q <= 0 or q != int(q) or p <= 0:  # Quantity must be a positive integer
            continue
        if abs(p * q - amount) <= 0.02:      # Split must match the amount
            item["Quantity"] = str(int(q))
            item["UnitPrice"] = rest
            return

Two details worth mentioning:

The tolerance is 0.02, because amounts have two decimal places and rounding introduces ±0.01 error.
Quantity must be a positive integer (q != int(q) skips it). Invoices do not show 1.5 bottles.

Also, the original precision of the unit price is deliberately preserved (294.247787610619), not rounded to two decimals—because downstream checks like unit price × quantity = amount would fail after truncation.

Hard problem 3: The "total" row is picked wrong when there is more than one page

This is the easiest place to bury a landmine, and single-page testing will never catch it.

On multi-page lists in full e-invoices, each page has both a subtotal and a total row at the bottom:

小  计   ¥415.29      ¥54.00        ← subtotal for this page
合  计   ¥82963.07    ¥10785.11     ← total for the entire invoice

If you pick the first row containing "合计", it will hit 小计 (because "小计" contains "计") and you get the single-page amount. On a 73-page invoice, the total becomes one four-hundredth of the real value.

So the locating logic is:

def find_total_row(rows, start):
    subtotal = -1
    for idx in range(start, len(rows)):
        row = rows[idx]
        if any(row.contains(w) for w in _EXCLUDE_TOTAL):
            continue

        has_amount = any(RE_AMOUNT_ONLY.match(nfkc(i.text).replace(" ", "")) for i in row.items)
        next_is_tax_total = idx + 1 < len(rows) and rows[idx + 1].contains("价税合计")

        # The next row is "价税合计" → this row must be the whole-invoice total, even if the keyword is broken into just "计"
        if next_is_tax_total and (row.contains("合计") or _is_summary_fragment(row)):
            return idx
        if row.contains("合计") and (has_amount or next_is_tax_total):
            return idx
        if row.contains("小计") and subtotal < 0 and has_amount:
            subtotal = idx          # Remember it, only fall back if no total row is found

    return subtotal

There is also a second case: OCR fragments "合计" so badly that only a single "计" character remains next to the amount:

计   ¥8849.56   ¥1150.44
价税合计（大写）壹万圆整    （小写）¥10000.00

In that case the character "计" alone is not enough, but the next row is "价税合计"—this structure is fixed on the invoice. Using it as a reverse proof confirms that the previous row is the total row.

After this logic worked, the 73-page PDF's line-item amount total 82963.07 and tax total 10785.11 matched the invoice exactly.

Hard problem 4: The full e-invoice header is artistic font

On full e-invoices (the ones with the 20-digit number), the header "电子发票" is in an artistic font. OCR often reads it as "国统", "电子发", or even "固统".

But there is one definite piece of information: the 20-digit number is unique to full e-invoices (paper invoices have 8, 10, or 12 digits). It can be used to correct the header:

def _normalize_full_e_invoice_name(self):
    number = self._get("发票号码")
    name = self._get("发票名称")
    if len(number) != 20 or not name or "电子发票" in name:
        return
    if "增值税专用发票" in name:
        self._values["发票名称"] = "电子发票(增值税专用发票)"
    elif "普通发票" in name:
        self._values["发票名称"] = "电子发票(普通发票)"

A single len(number) != 20 line excludes paper invoices and avoids false positives.

This pattern of "using one stable feature to correct another unstable field" is later applied to province inference and invoice-type classification, and it is more reliable than memorizing layouts.

Measured results

Metric	Result
Consecutive recognition	73 pages PDF, all 73 succeeded
Line-item amount total	82963.07 (matches invoice)
Line-item tax total	10785.11 (matches invoice)
Reconciliation difference	0.00
Total time	275 seconds (4-core CPU)
Per-image time	2–4 seconds

A recursive field-by-field comparison with Tencent Cloud VatInvoiceOCR's response (expanded to leaf nodes): 128 fields matched, 0 missing. The remaining differences are in coordinate scale, angle precision, and full-/half-width commas—none affect field values.

Known limitations

Written to save you time, not as modesty:

Only VAT invoices are covered (special, normal, electronic, full e-invoice). Train tickets, taxi receipts, bank slips are not handled yet and will fall into OtherInvoice.
For multi-page PDFs use the /ocr/vatpdf endpoint, not EnableMultiplePage. The latter runs all pages in one pass and will inevitably hit the reverse proxy's read timeout (default 60s).
PyMuPDF is AGPL-3.0. Source distribution is fine, but if you package it as closed-source binary for sale, you need an Artifex license or switch to pypdfium2.
The layout of full e-invoices is still being adjusted by the tax bureau, so new rules may need to be added when a new layout appears.

Repository

https://gitee.com/li_shuai_carson/invoice-ocr

git clone https://gitee.com/li_shuai_carson/invoice-ocr.git
cd invoice-ocr && docker compose up -d

The model is in the pip package, and after the image is built, the whole thing can run completely offline. MIT license, free for commercial use.

If you are using it, feel free to mention which invoice types you handle—toll invoices and rolled ones are not yet adapted, and those two layouts differ a lot from ordinary invoices.