AI Code Review Recall Put to a Blind Mutation Test: 13 of 20 Defects Caught
AI Code Review Recall Measured: 13 of 20 Mutants That Passed Unit Tests Caught
Last time, the focus was AI code review precision: 1,919 lines of Java, 18 findings, 8 true, 8 false, 2 pending, a 50% precision rate.
That answered only "does it report accurately?" This one answers: does it miss things?
The method is not "get a few people to rate their feelings" — it injects known defects into the code, runs a blind review, and counts hits. Design, results, reasons for misses, and costs are all laid out below; the method can be copied directly.
Conclusion
- 23 mutants injected; 3 were caught and eliminated by existing unit tests; 20 went into blind review
- Blind review caught 13, recall rate 65%
- Concurrency 4/4 all caught; security 3/4; null, exception, resource 2/4 each
- Of the 7 missed, 5 were "removing or swapping one defensive step"
- 20 blind runs cost about 1.65 million tokens; each run 50k–154k
Why Not Test Real Defects
The ideal corpus is production incidents and human review records. Not available: the project is a personal project, all 25 commits were self-made, there is no review feedback, hence no ground truth.
As a fallback, the mutation testing approach is used: inject defects yourself, be the judge yourself. The trade-off is that the distribution of mutants differs from real defects; conclusions can only indicate "sensitivity to this type of problem," not real recall rates. This point will be brought up again later.
Experiment Target
AI Agent Lab, a Spring AI practice project. Java 17, Spring Boot 4.1.1, Spring AI 2.0.1, DeepSeek, Spring MVC + SSE, Caffeine, ONNX Runtime. src/main/java has 34 files, 1,919 lines, and 34 unit tests all green.
Baseline snapshot tagged review-baseline (commit c5477f4).
Experiment Design
1. Mutants: 5 categories × 4 each
Five defect categories, 4 each, each change modifies one place, and must be applicable in this project:
| Category | Example | Candidate locations |
|---|---|---|
null-boundary |
reading first element of empty collection, out-of-bounds read, unboxing NPE | RagService topK, LexicalReranker empty candidate |
concurrency |
non-thread-safe collection, check-then-act, shared mutable state | BoundedChatMemoryRepository, ModelFileCache |
exception |
swallowing exception, wrong error code mapping, over-broad catch | ApiExceptionHandler, ChatService |
resource |
stream not closed, SSE not released, wrong close order | ChatController, RagService |
security |
logging PII / signed URL, unvalidated input | ChatController, ModelFileCache |
Hard constraints: mutants only go into src/main/java, tests are not modified; each has its own branch exp/mutants/M-XX, branched from review-baseline, never merged back to main.
2. Filtering Out "Too Easy" Ones with Tests
After applying each mutant, run ./mvnw test. Tests going red disqualify the mutant; a more subtle one is added in the same category.
Result: 3 of 23 were caught, all in the security category.
| mutant | change | why caught |
|---|---|---|
| M-18 | error event sends error.getMessage() instead of generic message |
assertion validates external error message |
| M-20 | remove hasText check in ConversationIds |
empty session ID test case exists |
| M-20R | out-of-bounds path concatenation | path traversal test case exists |
Two replacements were added: M-18R (logs full order table), M-20B (prints user message at streaming entry).
Existing tests only actively catch the ones with ready-made assertions. The 20 that entered blind review are exactly the areas tests don't cover; AI has meaning in filling that gap.
3. Blind Review Protocol
- Use fixed seed (20261027) to shuffle 20 mutants into B-01...B-20; mapping table stored separately, not visible during review.
- Review input only contains the mutated file, plus the interface signatures it directly depends on; no branch names, commit messages, or change descriptions.
- Review prompt frozen at v1, identical to the previous baseline review, without hints like "is there a bug here."
- Model fixed (claude-sonnet), no changes mid-way.
- Inject all, then shuffle, then review in one batch — not one-by-one, to avoid later runs being influenced by earlier ones.
- After judging finishes, compare with mapping table and record
caught_by_ai.
A gap that must be stated: the CLI does not expose the temperature parameter; the original plan to run the same module 3 times for variance was dropped. Hence there is no repeated-run consistency data below.
Results
Individual Judgments
| blind | mutant | category | mutation | hit |
|---|---|---|---|---|
| B-01 | M-03 | null-boundary | remove empty query protection | ✗ |
| B-02 | M-14 | resource | publish session before tokenizer | ✓ |
| B-03 | M-10 | exception | retryable error 503 → 502 | ✗ |
| B-04 | M-13 | resource | don't close knowledge base input stream | ✓ |
| B-05 | M-06 | concurrency | publish empty vector store first | ✓ |
| B-06 | M-11 | exception | non-200 branch doesn't clean up .part |
✓ |
| B-07 | M-18R | security | log full order table | ✓ |
| B-08 | M-09 | exception | broaden catch to Exception |
✓ |
| B-09 | M-15 | resource | close() set null before closing |
✗ |
| B-10 | M-08 | concurrency | delete double-check inside lock | ✓ |
| B-11 | M-12 | exception | remove IllegalStateException catch |
✗ |
| B-12 | M-01 | null-boundary | remove empty candidate early return | ✗ |
| B-13 | M-04 | null-boundary | days >= 0 → days > 0 |
✓ |
| B-14 | M-19 | security | append full signed uri to log |
✗ |
| B-15 | M-16 | resource | delete onCompletion cleanup |
✗ |
| B-16 | M-17 | security | chat() prints user message |
✓ |
| B-17 | M-07 | concurrency | remove failure backoff read | ✓ |
| B-18 | M-20B | security | streaming entry prints user message | ✓ |
| B-19 | M-02 | null-boundary | position <= candidates.size() |
✓ |
| B-20 | M-05 | concurrency | store caller's mutable list | ✓ |
By Category
| Category | hit / total | recall |
|---|---|---|
concurrency |
4 / 4 | 100% |
security |
3 / 4 | 75% |
null-boundary |
2 / 4 | 50% |
exception |
2 / 4 | 50% |
resource |
2 / 4 | 50% |
| total | 13 / 20 | 65% |
Analysis of Misses
7 missed, reviewed one by one.
1. B-12 Remove empty candidate early return (RagService:99)
Originally, empty candidate caused early return; the mutant deleted that block, so empty retrieval results continue through the LLM and the answer is fabricated. AI didn't mention it at all.
2. B-01 Remove empty query protection (LexicalReranker:33)
Empty query set leads to division by zero; scores become NaN and ordering becomes random. Not mentioned.
3. B-15 Remove onCompletion cleanup (ChatController:64)
After client disconnects, subscription is not released; the model keeps generating. Not mentioned.
4. B-11 Remove IllegalStateException catch (ChatController:156)
IllegalStateException thrown when the connection is closed is no longer swallowed and propagates upward. This is the most typical: AI discussed another issue in the same method (send fails without complete), showing the method was read, but the missing catch type wasn't noticed.
5. B-09 close() sets null before closing (CrossEncoderReranker:97)
Concurrent requests during the closing window get null or already-closed objects. Note: this has the same shape as B-05 (publish empty vector store first) which was caught; one caught, one missed — pattern recognition is not a stable rule.
6. B-03 Status code 503 → 502 (ApiExceptionHandler:56)
A retryable error gets mapped to non-retryable. Both numbers are valid in code; there is no "wrong" visible; the error is in the client's retry policy, which is not part of the input.
7. B-14 Log appends full uri (ModelFileCache:125)
Signed parameter goes into the log. The other three log leak cases were added as whole new print lines and all caught; appending a parameter to an existing log line was missed.
Summary
- 5 of the 7 misses were removing or swapping one defensive step: B-01, B-09, B-11, B-12, B-15. The code itself is completely valid; nowhere does it "look wrong."
- What was caught almost all has a "shape": lock position,
volatile, ordering ofsetandadd, new log lines, over-broadcatch. - It reviews the code written, not the code not written.
Another counterexample is numeric changes: B-13 (>= → >) was caught because the number was translated into semantics ("0 days being judged as already passed"); B-03 (503 → 502) wasn't, because the literal difference doesn't constitute an error. The clue is semantics, not numbers.
Cost
| Metric | Value |
|---|---|
| Blind token total | about 1.65 million |
| Per-run token range | 50k – 154k |
| Correlation between per-run cost and file size | weak |
Almost three times as expensive as baseline review (10 runs, 587k tokens), because run count doubled and each run carries full context. The expensive part is the file-reading rounds, not the code line count.
Limitations
- Single project, single judge: a ~2,000-line project, judged by one person; subjectivity cannot be removed.
- Mutants are artificially constructed: distribution differs from real defects. 65% only refers to sensitivity to this kind of injected defect, not true recall.
- Small sample: 20; a single hit is 5 percentage points; don't treat 65% as an exact value.
- The two metrics have different corpora: previous 50% precision from real code, this 65% recall from artificial injection; cannot synthesize an F1.
- No repeated runs: single run, no stability data.
Engineering Suggestions
- Give diffs, not just files. Deletion-type defects appear in a diff as "a few lines missing," which is visible; given the whole file, those lines simply don't exist.
- Don't rely on it for missing checks. Null checks,
finally, cleanup callbacks, exception branches — fall back to tests and diff review. - Use it for parts that are hard to test. Concurrency, resource, and exception-path tests are expensive; AI hit rate is highest there — this round concurrency 4/4.
- Only accept findings that map to line numbers, trigger conditions, and consequences. Whatever cannot be stated is noise.
- Let tests screen mutants first. What tests can catch goes to tests; AI only handles test blind spots, so both sides' accounts are clear.
Reproduction Clues
- Baseline: tag
review-baseline, commitc5477f4 - Mutants: 23 branches
exp/mutants/M-01...M-20B, one change per branch - Prompt: frozen
review-prompt-v1.md, three core disciplines — report only what can point to a line number, write clear trigger conditions and consequences, markneeds_contextfor judgments requiring unknown code and don't assume - Blind test: fixed seed shuffle, input only mutated file + directly dependent interfaces, map after judgment
- Cost logging: input/output token and duration per run
Full records (including prompt, raw output JSON, per-item judgment reasons) are under the project's docs/experiments/.
Data and judgments were manually verified by the author. This is a single experiment on a personal project; sample size and corpus have clear boundaries. Discount the conclusions according to your own engineering context.