Stop Trusting OCR-Specific Small Models: 8 Sub-7B VLMs Put to the Test on Structured JSON Extraction
For any team doing document digitization under data-residency constraints, this benchmark shows the cheapest reliable path is fine-tuning a general sub-7B VLM, not buying into OCR-specialized models. It also exposes that traditional image preprocessing can silently hurt modern VLM performance, so existing OCR pipelines need re-evaluation.
Museum digitization faces a hard constraint: documents like handwritten artifact registers can't be uploaded to commercial APIs, so institutions must run local models. Researchers at FAU Erlangen-Nuremberg tested 8 open-source vision-language models (200M to 7B parameters) across three real museum collections, evaluating zero-shot, few-shot, and fine-tuned performance on strict JSON schema compliance.
The results flip the common wisdom. OCR-specific models (Donut, PaddleOCR-VL) get perfect zeros on structured extraction metrics because their outputs can't even parse as valid JSON. General instruction-tuned models, especially Qwen2.5-VL-7B and Phi-3.5-Vision, win after fine-tuning, with Qwen reaching 90+ mAP-F1 on printed cards. The paper introduces mAP-F1, a metric that treats JSON parse failures as zero, cleanly separating content recognition from structural adherence.
Further experiments show classic image preprocessing (CLAHE, denoising) often harms modern VLMs, hyperparameter optimization helps but needs validation, and joint training across collections is model-dependent: Gemma-3 improves, Qwen2.5-VL degrades. The practical takeaway is that fine-tuning a general VLM on a single consumer GPU beats relying on OCR-specialized models for structured output.
The zero scores for OCR-specialized models expose a design gap: optimizing for character recognition does not produce schema-compliant structured output, and the interface limitations of those models make them useless for JSON-oriented tasks.
The anti-preprocessing results are counterintuitive but logical: modern VLMs pretrain on raw distributions, so classic OCR enhancement steps push inputs outside the learned distribution, negating their intended benefit.
Joint training outcomes being so model-dependent (Gemma improves, Qwen degrades) suggests that catastrophic forgetting and negative transfer are not predictable from model size or family alone, forcing empirical evaluation per deployment.
The small dataset sizes and single-run metrics mean the leaderboard gaps, especially those within a few points, are likely noise; the qualitative improvements after fine-tuning are more trustworthy than precise rankings.
Leaving constrained decoding as future work is a missed opportunity since it directly addresses the parse-failure failure mode, and could potentially revive OCR-specialized models for structured tasks.