A Weekend Plate Recognizer Hits 99% on CCPD, Then Real Photos Humble It
Benchmark accuracy on a curated dataset like CCPD can mislead; the long tail of real-world captures—province distribution, camera angle, lighting, and document reproduction—is what actually determines production performance, so teams should evaluate by subgroup rather than a single aggregate metric.
A two-day DIY build produced a license plate recognition pipeline: YOLOv8n for detection, LPRNet with CTC for end-to-end character recognition, and a small FastAPI service wrapped around it. On the CCPD benchmark, the model hit 0.99 mAP50 for detection and strong exact-match recognition, thanks largely to CCPD's clean, front-facing, Anhui-dominant imagery.
When the same pipeline met real vehicle photos from an auto insurance business, accuracy collapsed. Out-of-province plates were guessed wrong, tilted and distant shots missed plates entirely, double-deck yellow plates became garbled, screenshots and PDF scans never appeared in training, and HSV color thresholds flipped blue to black at dusk. The post's lesson is blunt: model structure sets the floor, data sets the ceiling. The long tail of real captures—province mix, camera angle, lighting, and document reproduction—dwarfs model choice.
The developer lists practical fixes: add data by province and color, augment for real capture conditions, evaluate accuracy separately by subgroup rather than a single total, and replace the HSV color heuristic with a learned classifier. The business integration quietly went back to the commercial API that had worked all along.
The gap between 99% on CCPD and poor real-world performance is a distribution shift—CCPD is mostly front-facing, clear, Anhui-registered vehicles—so the model architecture, YOLOv8n plus LPRNet, is not the weak point; the data coverage is.
A single aggregate accuracy number masks large variances per subclass; the evaluation strategy of splitting by province, color, and capture condition is what separates a demo from something deployable.
The author's experience with HSV color thresholds shows how fragile hand-tuned rules are under lighting changes; a learned classifier is a cheap and reliable upgrade.
The commercial API the developer switched back to likely succeeds not because of a fancier model, but because it has trained on a much broader distribution of real-world captures over time.