Stop Trusting OCR-Specific Small Models: 8 Sub-7B VLMs Put to the Test on Structured JSON Extraction
TL;DR: Museum digitization has a catch-22: documents cannot be uploaded to commercial APIs, but local small models can't extract qualified structured data. The German FAU team tested 8 open-source models under 7B on three real collections (handwritten artifact registers, jewelry cards, staircase survey forms). Three counterintuitive conclusions: ① OCR-specific small models almost all fail at structured extraction (JSON synthesis rates too low); fine-tuned general instruction models are the winners. ② Classic image preprocessing (denoising, CLAHE) often hurts on modern VLMs. ③ When training on multiple collections jointly, some models improve (Gemma-3) while others get significantly worse (Qwen2.5-VL). Fine-tuning is the most effective strategy and runs on a single consumer GPU.
Have you ever thought that those yellowed registers in museums are actually data gold mines?
The excavation site, dimensions, and description text of a prehistoric stone artifact; the material, weight, and insured value of a 19th-century piece of jewelry: if this information became structured data, researchers could instantly search through a century of collections. The problem: these registers are handwritten in German, the layout is over a hundred years old, and, most critically, they are institutional assets that cannot simply be uploaded to commercial APIs.
That is the reality the team from Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) faced in this paper. Their answer wasn't to train a bigger model, but a solid benchmark: 8 open-source vision-language models (VLMs) under 7B, 3 real collections, all tested from zero-shot, few-shot, to fine-tuning. The results are quite interesting.
Paper: arXiv:2610.11818
Code: github.com/uddipan77/Analysis-of-Lightweight-Vision-Language-Models
The task isn't "reading characters", it's "reading characters + filling out a form"
Make the task clear first: given a document image, the model must output a JSON object that strictly conforms to a predefined schema. Not free text, not Markdown, but structured data that can be directly fed into an institutional database and validated programmatically.
Look at a real sample: an index card from the FAU prehistoric collection.
On this card, the header is printed ("Anthropologisch-prähistorische Sammlung..."), the body is handwritten German cursive, and there is a stone tool sketch in the lower left. The model must extract fields including: inventory number (J.-Nr. Eo 2), dimensions (L 7,25 B 4 cm D 4,2), excavation site (Spy, Belgien), collection date, full description, and so on: 8 fields, no misalignment allowed.
The dataset has three sets with very different styles:
| Dataset | Count | Text type | Schema complexity |
|---|---|---|---|
| Erlangen prehistoric artifact register | 308 | mostly handwritten | medium (8 fields) |
| Pforzheim jewelry catalog cards | 590 | mostly printed | medium (17 fields) |
| Regensburg staircase survey forms | 164 | mixed | high (deep nesting + boolean fields) |
The third one is the most brutal. The Regensburg "scalalogy" (Scalalogy, a real discipline) form has nested objects and dozens of boolean fields: spiral vs. straight stairs, wooden vs. stone, which side the railing is on. This is a stress test for hierarchical schema compliance.
The jewelry card looks like this:
Eight contestants, two "competition formats"
The 8 models in the benchmark range from 200M to 7B parameters:
- Fixed-prompt group (inflexible interface, only fixed OCR instructions): Donut-Base, Donut-CORD (each 200M), Florence-2-Large (0.77B), PaddleOCR-VL (0.9B)
- Custom-prompt group (can take natural language instructions): Nanonets-OCR-s (4B), Gemma-3-4B, Phi-3.5-Vision (4.2B), Qwen2.5-VL-7B
There is an easily overlooked detail here, and it is the most methodologically careful part of this paper: constraint-aware protocol. The two groups have completely different interface capabilities; the former cannot even say "output JSON according to this schema", so you can't force the same prompt on both. The paper's approach: let each model compete under its supported interface, with only minimal post-processing for the fixed-prompt group. That is a fair answer to the question "can lightweight models be used?"
The evaluation metrics are also worth elaborating. Besides the usual CER (character error rate) and ANLS* (structural similarity), the team proposes mAP-F1: flatten the predicted and ground-truth JSON to leaf keys, compute per-field overlap as 1-CER, and take the macro average over 4 thresholds. The key rule: if JSON parsing fails, score is 0. This design separates "content recognition accuracy" from "structural output correctness", which exactly exposes the gap between the two model groups.
Results: OCR-specific models crash
Zero-shot scores, mAP-F1 (higher is better, three datasets):
| Model | Prehistoric (handwritten) | Jewelry (printed) | Stairs (nested) |
|---|---|---|---|
| Donut-Base | 0.0 | 0.0 | 0.0 |
| PaddleOCR-VL | 0.0 | 0.0 | 0.0 |
| Nanonets-OCR-s | 28.5 | 86.5 | 15.2 |
| Gemma-3-4B | 34.6 | 87.4 | 22.5 |
| Qwen2.5-VL-7B | 56.2 | 96.3 | 33.3 |
Yes, you read that correctly. Donut-family and PaddleOCR-VL score 0 on all structured metrics. Not because they can't recognize characters, but because their outputs cannot be parsed into valid JSON (the paper discloses that even after fine-tuning, their JSON synthesis rates remain very low). OCR-specific models are top students in "plain text transcription", but rigid interfaces and missing结构化 capabilities make them abstain in this track.
Qwen2.5-VL-7B leads zero-shot across the board, hitting 96.3 mAP-F1 on jewelry cards: printed text plus flat schema is near saturation. But handwritten prehistoric registers (56.2) and nested stair forms (33.3) have much room for improvement.
After fine-tuning, the picture is clear. Top-3 are all general instruction models: Qwen2.5-VL-7B (90.0 / 99.4 / 62.8), Phi-3.5-Vision (82.2 / 99.5 / 72.0, best on stairs), Gemma-3-4B (83.1 / 99.4 / 46.9).
The paper's summary is on point: reliable structured extraction depends not only on "recognizing characters accurately", but also on instruction following, field binding, and stable output of schema-compliant content, which is exactly the home turf of general instruction models.
Here's the hard stair survey sample, feel the "deep nesting":
Four "enhancement check-ups": two often backfire
After selecting the top 3, the team ran three sets of controlled experiments, each isolating one contribution. This part may be more valuable to practitioners than the leaderboard itself.
① Hyperparameter optimization (Optuna) — the most reliable gentle enhancement. All three models improve on stairs (Gemma-3's mAP-F1 jumps from 46.9 to 77.9). But note the word "gentle": not a free lunch, since Gemma-3 actually drops on prehistoric from 83.1 to 65.4. Conclusion: HPO is worth doing, but validate on the validation set afterwards, don't assume guaranteed gains.
② Classic image preprocessing — mostly harmful. Illumination flattening, denoising, CLAHE, those standard operations from the traditional OCR era, often degrade performance on modern VLMs: Qwen2.5-VL on stairs 62.8→53.6, Gemma-3 from 46.9→43.6. Occasional CER improvements don't carry over to structured metrics. Intuitive explanation: these models were pretrained on "raw" image distributions; your carefully PS'd image actually moves it out of its comfort zone.
③ Multi-stage training — purely model-dependent. Gemma-3 improves on all three datasets, Phi-3.5 fluctuates slightly, Qwen2.5-VL degrades on two (jewelry 99.4→90.2). Curriculum learning is not a silver bullet.
Joint training: one checkpoint for all collections?
The ideal deployment for an institution is a single model serving all collections. The paper tests "single-dataset specialized checkpoints" vs. "a general checkpoint trained on all three collections".
Once again the results split:
- Gemma-3-4B: joint training improves everything — prehistoric 83.1→87.9, stairs 46.9→68.0. Cross-schema training actually boosts its generalization.
- Qwen2.5-VL-7B: clear negative transfer — stair CER worsens from 7.6 to 24.9, mAP-F1 drops from 62.8 to 41.4.
The same operation helps one model and hurts another, and both by non-trivial amounts. The paper's position is restrained: there is no universal rule; institutions must test with their own models and data before deployment — either maintain one general checkpoint or specialize per collection, no obvious answer.
What does fine-tuning actually improve? Look at the paper's qualitative comparison: after fine-tuning Gemma-3, the inventory number format binding is correct (J.-Nr. E024 → Eo 24), the dimension fields go from three empty strings to 6,25 / 3,95 / 1,7, and per-sample CER drops from 26.8% to 13.9%. But long handwritten descriptions and field-boundary confusion between excavation site and collection date remain unresolved.
Cold water time
This paper is solid, but several points deserve closer scrutiny:
Small datasets. The staircase test set has only 25 images, jewelry 89. At this scale, a few percentage points may fall outside statistical significance, and the paper doesn't report variance across multiple runs. Treat the leaderboard details cautiously.
Part of ground truth comes from Gemini. The prehistoric register transcriptions were generated by Gemini and manually validated. The quality of validation (e.g., inter-annotator agreement) is not disclosed — although the data is released on Zenodo for re-checking, the "AI-annotated + human-reviewed" loop itself is worth noting.
No constrained decoding tested. For the pain point of "JSON parse failure = 0", the industry has constrained decoding methods that force valid JSON output. The paper only lists it as future work. If added, would the OCR-specific models' zero-score disaster change? Hard to say, but it's a reasonable question.
Time-sensitive. Models iterate too fast (PaddleOCR-VL is rapidly updating), and the paper itself acknowledges conclusions apply only to the tested versions.
Rational takeaway
Zooming out, this paper answers a "last mile" problem: when large model capability is overflowing, how can institutions use it well under data sovereignty constraints.
Three practical conclusions to take away:
- Choose models by task, not by track label. "OCR-specific" does not mean "document extraction-specific". For structured output, prioritize general VLMs that support custom instructions.
- Fine-tuning is the main strategy, HPO is a safe enhancement, while "traditional wisdom" like image preprocessing and multi-stage training should be validated before applying, not assumed beneficial.
- Test joint training per model. Gemma-type models can eat and grow fat, Qwen-type models get indigestion.
For teams doing enterprise document digitization and archival systems in China, this report is almost a deployment pitfall checklist. In scenarios where data cannot leave the premises, sub-7B models plus a consumer GPU (sub-1B full fine-tuning, 4-7B QLoRA) can already reach "usable" level — unthinkable two years ago.
As for the remaining points — field boundaries in long handwritten text, forced-JSON decoding strategies — leave them for the next batch of papers.
(This is article #13 in the "Intelligent Documents" series. Related reading: EVICALC: Why LLMs fail at document arithmetic, TRACE: The contract gap in document evidence localization, FoveDoc: Retrieval augmentation fills the reading gap.)