A Private AI Wrong-Question Notebook Closes the Loop from Photo to Practice Paper

AI Wrong-Question Notebook: A Complete Closed Loop from Photo → Human Review → Practice → Paper Generation

1. Why I built this

My child is in junior high school, and math practice books and test papers keep coming in waves. The traditional wrong-question management process involves copying questions (time-consuming), pasting papers (messy), and searching through piles when reviewing. The photo-search apps on the market only give answers; they don't address "where did I go wrong on this question, and will I make the same mistake again next time?"

What I wanted was a private wrong-question knowledge base:

• Take a photo to digitize an entire page of questions; I review which ones are worth storing and what the error cause is.
• Normally I can practice questions and generate papers by knowledge point.
• Questions answered correctly three times in a row automatically get suggested for archiving, so truly mastered questions stop taking up attention.
• The data stays in my own hands.

Coincidentally, I had previously built a question-generation system (quiz-demo) that contained a battle-tested "quality chain": geometry numeric validation (geocheck) → independent solving verification (verify_answers) → vector paper export (export_paper). The wrong-question notebook directly reuses this chain: all AI-generated questions must pass these three steps; no naked generation.

2. Complete workflow

Upload photos (multiple allowed)
  → Visual model transcribes multiple questions (stem LaTeX, options, graphics bounding boxes, printed answers, handwriting marks)
  → Automatically crop original images by bounding boxes (authoritative graphics, 0 tokens)
  → [Human review view] left original image, right question card, KaTeX real-time rendering
      Check to store + mark error cause (careless/don't know/other) + graphics can use crop or manual box selection
      Grade and knowledge point auto-categorized (textbook knowledge tree), editable
  → Submit to bank: dedup against question bank → independent solving verification → chapter/difficulty/knowledge-point analysis
  → Workbench (filter/statistics/details)
  → Practice (single-question loop, auto-grading) / Exam (paper mode with checkboxes, unified grading with score + evaluation)
  → 3 consecutive correct → suggest archive → archive
  → Targeted paper generation (by chapter/knowledge point/difficulty ratio, original question variants or original) → vector PDF

3. Features list (V1.0)

1. Recording & typesetting machine (the most labor-intensive module)

• Multi-question transcription: All questions in one photo are transcribed at once; only printed text is recognized, handwriting traces are only marked, not transcribed.
• Human review first: Transcription results must go through human review (left original, right question card for comparison); only checked questions are stored. AI only drafts; human decides.
• Graphics source layering: Geometry questions can generate a standard vector SVG with one click (geocheck numeric validation loop); the default is the original photo crop (saves tokens); table and function-graph questions must use the original image.
• Three graphics correction entry points: Re-fetch original image, manual drag-and-drop box selection (box archived and replayable), and whole-form idempotent re-crop.
• EXIF orientation unified: Phone photo rotation info once caused crops to rotate 90° and manual box selection to be completely misaligned; unifying the entire pipeline to an "upright space" fixed it.
• Grade/knowledge-point auto-categorization: The textbook (Zhejiang Education version, grade 8, book 1) was transcribed into a knowledge base (201 pages) using build_kb; at input time, the LLM can only select within the knowledge tree; human review can adjust.

2. Quality chain for entering the bank

• Deduplication: After normalizing the question stem (removing LaTeX command names, punctuation, and whitespace), similarity ≥92% is considered a duplicate and it is not re-entered; reported in the completion banner.
• Don't trust photo answers: The printed answer in the photo is only a reference; another model always independently solves the problem to cross-check. Conflicts are resolved by the audited answer, with a trace left; questions that fail verification go into needs_review.
• Error-cause-driven analysis: Careless → the likely-erroneous step is recorded as "easy-to-miscalculate part"; don't know → recorded as "thinking blockage point".

3. Workbench

• Statistics charts (total / in bank / archived / zero practice / suggested archive) plus five distribution charts (collapsed by default, click to view).
• Wrong-question list: formula rendering, columns for chapter/difficulty/knowledge point/error cause, and a view popup (full question info + each practice history).
• Linked filters: chapter/difficulty/error cause/knowledge point/include archived; after a search, statistics and list update together; filters and selections persist across tabs.
• Inline archive/activate; after checking, you can directly export a practice paper PDF (vector formula typesetting, answers at the end of the paper, cuttable).

4. Practice and exam (two modes)

• Practice (single-question mode): Automatically loops through all questions in the bank; single-choice, multiple-choice, and fill-in-the-blank are auto-graded, open-ended questions are self-judged against the answer. Immediate feedback includes consecutive correct n/3, archive suggestions, repeated mistakes, and the option to do similar questions.
• Exam (paper mode): Check questions in the workbench → answer the full paper in a new tab → unified grading: score (out of 100), evaluation, and a wrong-question knowledge-point summary (pointing directly at weak spots).
• Similar questions: Generated around the same knowledge point and error-prone point; they pass graphics validation plus verification; if they fail, they are automatically regenerated.

5. Multi-user

Storage is per-user with separate directories (question bank / drafts / photos), with header-based switching/creation of users and automatic migration of existing data. Two children's wrong questions don't interfere with each other.

4. Several key design decisions

1. Data as files, not SQLite

All data is JSON (atomic writes + .bak backup), human-readable, and git-versionable (the question bank itself is an important asset). At the scale of a few hundred questions, a single git log is the complete evolution history. Later, a test accident overwrote the question bank, and it was exactly the backup and git that recovered it.

2. Vectorization (SVG generation) changed from default to on-demand

The initial design was to automatically convert all geometry questions into standard vector figures. But testing showed this is the most token-consuming and slowest step in the entire pipeline (1–3 LLM calls per question plus geocheck failure retries). Version 0.6 flipped it: transcription defaults to photo crops (0 tokens), and standard graphics are generated on demand during human review. This saves more than half the overhead and greatly speeds up input.

3. Don't let the LLM freely choose knowledge points

Knowledge-point candidates come from the textbook knowledge base (a paper textbook is visually transcribed page by page into structured JSON). At input, the LLM can only select within the tree. Meanwhile, the knowledge-point candidates used for workbench filtering come from actual labels in the question bank, ensuring that whatever you select in the dropdown is searchable (the mismatch between textbook naming and LLM annotation naming is a real pitfall).

4. Human review is the product's core, not a transitional plan

The uncertainty of photo transcription (multi-question layouts, handwriting interference, print quality) makes fully automatic entry unreliable. Rather than chasing 'full automation', it's better to make the human review interface smooth: compare with the original image, check each question, mark error cause, and correct graphics, all within two clicks. AI handles 80% of the grunt work, and humans handle 20% of the judgment.

5. Pitfalls record (selected)

• CV-based graphics location fails on real photos: Line-profile positioning achieved IoU 95.9% on fake photos, but on real photos it mistook slanted text for graphics. After two rounds of testing, it was abandoned in favor of having the vision model directly output graphics bounding boxes.
• DeepSeek's thinking chain burned through max_tokens: This left content empty with finish_reason=length. The degradation strategy was json_object → turn off thinking and retry → plain, which measured 645s → 16s.
• EXIF orientation dual coordinate systems: The browser auto-orients but the server doesn't, causing crops to rotate 90° and manual box selection to be completely misaligned. The fix was unifying the entire pipeline to the upright space plus a historical data migration tool.
• Formulas sometimes rendered and sometimes didn't in lists: Truncating the question stem to 40 characters cut off the $...$ pairing, so KaTeX gave up when it couldn't find the closing symbol. The fix was returning the full text and using CSS single-line truncation.
• Exporting PDF showed LaTeX source: The export pipeline didn't render LaTeX, so a LaTeX → Unicode math symbol conversion layer was added (\triangle→△, \circ→°, \frac→(a/b), superscripts to ⁰¹²³).
• Tests covered the real question bank: Tests patched path constants, but after a storage layer refactor the paths were rebound, and seed data overwrote real data (10 questions → 1). It was recovered via .bak and git, and test mode was added as insurance for all tests. This incident once again proved the value of 'data as files + git management'.

6. Next steps

• Periodic review reminders for archived questions (turning the wrong-question notebook into a review system).
• Mobile adaptation (the photo-taking scenario naturally happens on a phone).
• Extend the textbook KB to grade 9 and normalize knowledge points to textbook naming.
• Long-term trend analysis for error causes and knowledge points.

Conclusion

The biggest takeaway from this project is: the value of an AI application is not in the model itself, but in the engineering surrounding it—the quality chain, the human review process, data management, and test regression. These seemingly dull efforts determine whether it can actually be used every day. Three days of development plus 12 iterations driven by real usage feedback, and it's now a tool my family uses daily.

---

This project is a self-use tool; the server deployment is not open-sourced. If you're interested in the quality chain (geocheck / verify_answers / export_paper) or the details of the wrong-question notebook implementation, you're welcome to discuss.