A Weekend Plate Recognizer Hits 99% on CCPD, Then Real Photos Humble It

前言

It goes like this.

The company does auto insurance business. When recording a policy, we have to extract the plate number from vehicle photos. We've been using Bluexing AI's license plate recognition API: send an image, and within a few hundred milliseconds it returns the plate number, color, and single- or double-deck type. It's been dependable.

But as a curious programmer, I kept thinking: "Isn't it just plate recognition? YOLO everything—how hard can it be?"

So one weekend I decided to build one from scratch, end to end.

Spoiler for the ending: The model ran, the metrics looked great, and then it got humbled on real photos. 😅

This post records the full two-day process. All the code runs directly, and I'll talk through the pitfalls I hit.

---

🎯 Set a Target

Input a vehicle photo, output:

{
  "plateNumber": "川AQ3C28",
  "plateColor": "blue"
}

Breaking it down, there are really three steps:

1. Find where the plate is: object detection
2. Recognize the characters on the plate: text recognition
3. Determine the plate color: blue, green, yellow...

I didn't agonize over the tech stack—I went straight to the most classic, most well-documented combination:

Step	Approach	Why I chose it
Detection	YOLOv8n	Light, fast, well-documented, pip install and it works
Recognition	LPRNet	No character segmentation, CTC end-to-end, the model is only a few hundred KB
Color	HSV statistics	Get something working first, optimize later

The overall pipeline looks like this:

---

📦 Day 1 Morning: Data

The most common open-source dataset for license plate recognition is CCPD, with hundreds of thousands of images covering blue and green plates.

The first time I opened the dataset, I was stunned: No annotation files?

Looking closely, the annotations are all encoded in the filenames:

025-95_113-154&383_386&473-386&473_177&454_154&383_363&402-0_0_22_27_27_33_16-37-15.jpg

Pretty crude, right 😂. Split by -; segment 3 is the plate bounding box coordinates, segment 5 is the plate number encoding.

I wrote a script to convert to YOLO format:

import os
import cv2

def ccpd_to_yolo(img_dir, label_dir):
    os.makedirs(label_dir, exist_ok=True)
    for name in os.listdir(img_dir):
        if not name.endswith(".jpg"):
            continue
        box = name.split("-")[2]
        (x1, y1), (x2, y2) = [map(int, p.split("&")) for p in box.split("_")]
        img = cv2.imread(os.path.join(img_dir, name))
        if img is None:
            continue
        h, w = img.shape[:2]
        cx, cy = (x1 + x2) / 2 / w, (y1 + y2) / 2 / h
        bw, bh = (x2 - x1) / w, (y2 - y1) / h
        with open(os.path.join(label_dir, name[:-4] + ".txt"), "w") as f:
            f.write(f"0 {cx:.6f} {cy:.6f} {bw:.6f} {bh:.6f}\n")

I also parsed out the plate number for later training of the recognition model:

PROVINCES = ["皖", "沪", "津", "渝", "冀", "晋", "蒙", "辽", "吉", "黑", "苏", "浙", "京", "闽", "赣", "鲁",
             "豫", "鄂", "湘", "粤", "桂", "琼", "川", "贵", "云", "藏", "陕", "甘", "青", "宁", "新"]
ALPHABETS = list("ABCDEFGHJKLMNPQRSTUVWXYZ")
ADS = list("ABCDEFGHJKLMNPQRSTUVWXYZ0123456789")

def ccpd_plate_text(name):
    idx = list(map(int, name.split("-")[4].split("_")))
    return PROVINCES[idx[0]] + ALPHABETS[idx[1]] + "".join(ADS[i] for i in idx[2:])

💡 Fun fact: The letters I and O are absent from license plate alphabets, because they're too easily confused with 1 and 0; the national standard forbids them. No need to include them in the character set.

---

🏋️ Day 1 Afternoon: Train the Detection Model

YOLOv8 really lowered the barrier to the floor. Write a config file:

# plate.yaml
path: datasets/plate
train: images/train
val: images/val
names:
  0: plate

Then train with one command:

yolo detect train data=plate.yaml model=yolov8n.pt epochs=100 imgsz=640 batch=32

Went to eat; came back, and mAP50 was already 0.99.

Me at that moment: 🤩 This is too easy!

(Remember that emoji, it'll matter later)

Export to ONNX for deployment later:

yolo export model=runs/detect/train/weights/best.pt format=onnx imgsz=640

---

🔤 Day 2 Morning: Recognize Plate Characters

Detection just finds the plate; next, recognize the characters on it.

LPRNet's approach is elegant: Don't segment individual characters. Feed the whole plate image in, output a sequence where each position is the probability of "which character is here", then decode with CTC.

CTC decoding sounds high-level, but the greedy version is just two rules: drop blanks, merge adjacent repeated characters.

def ctc_greedy_decode(indices, charset, blank):
    out, prev = [], blank
    for i in indices:
        if i != blank and i != prev:
            out.append(charset[i])
        prev = i
    return "".join(out)

For example, model output 川川-AA-Q-33-C-2-8 (- is the blank), decoded becomes 川AQ3C28.

Why do blanks exist? Because a plate might have consecutively identical characters, like 京A11111; without blank separating them, 11111 would merge into a single 1.

LPRNet has ready open-source implementations; just train on the plate numbers parsed from CCPD. I won't expand further here.

---

🚀 Day 2 Afternoon: Wire It Up, Wrap It as an HTTP Service

Both models are ready; write an inference class. Only depends on onnxruntime and opencv, no torch needed:

import cv2
import numpy as np
import onnxruntime as ort

CHARS = list("京沪津渝冀晋蒙辽吉黑苏浙皖闽赣鲁豫鄂湘粤桂琼川贵云藏陕甘青宁新"
             "0123456789ABCDEFGHJKLMNPQRSTUVWXYZ学警港澳挂使领") + ["-"]
BLANK = len(CHARS) - 1


class SimpleLPR:
    def __init__(self, det_path, rec_path, conf=0.4, iou=0.5):
        self.det = ort.InferenceSession(det_path, providers=["CPUExecutionProvider"])
        self.rec = ort.InferenceSession(rec_path, providers=["CPUExecutionProvider"])
        self.conf, self.iou = conf, iou

    def detect(self, img, size=640):
        h, w = img.shape[:2]
        r = min(size / h, size / w)
        nh, nw = int(h * r), int(w * r)
        canvas = np.full((size, size, 3), 114, np.uint8)
        canvas[:nh, :nw] = cv2.resize(img, (nw, nh))
        x = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB).transpose(2, 0, 1)[None].astype(np.float32) / 255
        pred = self.det.run(None, {self.det.get_inputs()[0].name: x})[0][0].T
        pred = pred[pred[:, 4] > self.conf]
        if len(pred) == 0:
            return []
        boxes = np.stack([pred[:, 0] - pred[:, 2] / 2, pred[:, 1] - pred[:, 3] / 2,
                          pred[:, 2], pred[:, 3]], 1) / r
        keep = cv2.dnn.NMSBoxes(boxes.tolist(), pred[:, 4].tolist(), self.conf, self.iou)
        res = []
        for i in np.array(keep).flatten():
            x0, y0, bw, bh = boxes[i]
            res.append((max(int(x0), 0), max(int(y0), 0), int(x0 + bw), int(y0 + bh)))
        return res

    def recognize(self, crop):
        x = cv2.resize(crop, (94, 24)).astype(np.float32)
        x = ((x - 127.5) * 0.0078125).transpose(2, 0, 1)[None]
        idx = self.rec.run(None, {self.rec.get_inputs()[0].name: x})[0][0].argmax(0)
        text, prev = "", BLANK
        for i in idx:
            if i != BLANK and i != prev:
                text += CHARS[i]
            prev = i
        return text

    @staticmethod
    def guess_color(crop):
        h, s, v = [c.mean() for c in cv2.split(cv2.cvtColor(crop, cv2.COLOR_BGR2HSV))]
        if v < 60:
            return "black"
        if s < 50:
            return "white"
        if 100 <= h <= 130:
            return "blue"
        if 15 <= h <= 35:
            return "yellow"
        return "green" if 35 < h < 100 else "unknown"

    def __call__(self, img):
        out = []
        for x1, y1, x2, y2 in self.detect(img):
            crop = img[y1:y2, x1:x2]
            if crop.size:
                out.append({"plateNumber": self.recognize(crop), "plateColor": self.guess_color(crop)})
        return out

A bare class isn't satisfying enough, so I wrapped it with FastAPI so frontend colleagues can call it directly:

# server.py
import cv2
import numpy as np
from fastapi import FastAPI, File, UploadFile

from lpr import SimpleLPR

app = FastAPI()
lpr = SimpleLPR("models/plate_det_yolov8n.onnx", "models/lprnet.onnx")


@app.post("/plate")
async def plate(file: UploadFile = File(...)):
    data = np.frombuffer(await file.read(), np.uint8)
    img = cv2.imdecode(data, cv2.IMREAD_COLOR)
    if img is None:
        return {"code": 400, "msg": "图片解析失败"}
    return {"code": 200, "data": lpr(img)}

pip install fastapi uvicorn python-multipart onnxruntime opencv-python
uvicorn server:app --port 8000
curl -F "[email protected]" http://127.0.0.1:8000/plate

The whole flow looks like this (schematic):

Sunday evening: exact-match accuracy on the CCPD test set was beautiful. I thought I could wrap up.

---

💥 Then, the Crash

Monday at work, I pulled a batch of real vehicle photos from the business system to test.

Me at that moment: 🤡

Problems kept popping up:

1. Out-of-province plates all misclassified

CCPD was collected in Anhui, and most data is "皖" plates. The model basically only knows 皖; it started guessing randomly on 川, 粤, etc.

2. Phone tilt shots and distant shots missed detection

CCPD is mostly front-facing shots from parking lots. Business staff took photos at every possible angle, plus night, backlighting, and plate glare.

3. Double-deck yellow plates turned into gibberish

Double-deck plates common on trucks and trailers: LPRNet defaults to single-row handling and crashes.

4. Screenshots, re-photographed, and PDFs completely unseen

In car insurance material, a lot of it is phone re-photography of a computer screen, or PDF scans, with moiré patterns and compression noise. Not one such image in the training set.

5. Color judgment was a mess

The HSV thresholds work fine in normal light; shoot in the evening, and a blue plate gets judged black, a green plate yellow.

I ran the same batch of "hard images" through the Bluexing AI plate recognition API we normally use, and the gap was exactly on these scenarios. The official site has a web demo where you can upload directly, free 10 times a day, images and PDFs both supported. If you want to compare your own model, it's a good reference.

---

🤔 Retrospective: The Hard Part Is Never the Model

After all this, my biggest takeaway:

Model architecture sets the floor; data sets the ceiling.

YOLOv8 + LPRNet reaching 99% on an open dataset isn't hard. The hard part is:

What I thought was hard	What's actually hard
Which model to choose	Where to find data covering all provinces
Which hyperparameters to tune	How to cover tilted shots, night, re-photography, PDFs
How to get accuracy to 99%	How to handle long-tail cases: double-deck, HK-Macau, new-energy plates
How to speed up inference	Engineering problems: online concurrency, error codes, traceability

If you want to keep optimizing, here are a few directions:

• Add data: supplement real samples by province and color; if insufficient, synthesize with font rendering;
• Add augmentation: brightness/contrast, blur, rotation, perspective, JPEG compression, to simulate real capture conditions;
• Grouped evaluation: always compute accuracy separately by province and color; overall accuracy lies;
• Swap color for a classification network: far more stable than HSV thresholds.

---

📝 Finally

The weekend wasn't wasted. At least I thoroughly understand the detection, CTC recognition, and ONNX deployment chain. I strongly recommend anyone wanting to get into computer vision to work through it themselves.

As for the company's business? I quietly switched the API call code back to Bluexing AI 🙃. Because ticking off every item in that "actually hard" table is not something a weekend can handle.

The complete code is above. Copy it, add the model files, and it runs. Questions welcome in the comments; give a like if it helped 👍