AI Code Review Recall Put to a Blind Mutation Test: 13 of 20 Defects Caught
AI code review tools now have a measurable recall ceiling: they catch about two-thirds of injected defects but systematically miss deleted defensive code and semantic-only numeric changes, so they can't replace diff review or tests for those classes. The experiment also gives other teams a copyable protocol for benchmarking their own AI review on synthetic defects.
A single-developer experiment measures how often AI code review actually catches injected defects. The author seeded 23 mutants into a 1,919-line Java/Spring project, ran existing unit tests to disqualify any that failed, and sent the surviving 20 files to Claude Sonnet with a frozen review prompt and no hint that defects existed. The AI caught 13, a 65% recall rate. Concurrency defects were all caught (4/4); security hit 3/4; null-boundary, exception, and resource categories each landed 2/4.
The missed defects cluster into a clear pattern: 5 of 7 misses were deletions or substitutions of a single defensive step, such as removing an empty-query guard or a cleanup callback. The code still compiles and looks valid. The AI caught structurally "shaped" anomalies — lock positions, volatile, new log lines, over-broad catches — but remained blind to absent code. A numeric change like `>=` to `>` was caught only because it carried semantic meaning; a status code 503 to 502 change was missed because both numbers are valid in isolation.
The 20 blind runs consumed about 1.65 million tokens (50k–154k per run), nearly triple the cost of the earlier precision baseline. The author offers five engineering rules: feed diffs rather than whole files, use tests to pre-filter mutants, rely on AI for concurrency and resource-path gaps, accept only line-numbered findings, and never combine the two corpora into an F1 score.
The misses follow a consistent cognitive failure: AI review is sensitive to code patterns and structural anomalies but blind to the absence of defensive lines that would have been visible in a diff.
The 65% recall and the earlier 50% precision come from different corpora, so the author correctly refuses to combine them into an F1 score; this limits practical utility for teams wanting a single quality number.
A single run with no temperature variation means the 65% is a point estimate with no variance; the author explicitly flags this, which makes the result more trustworthy than many AI benchmarks.
The fact that the same shape of defect (publishing before initialization) was caught once and missed once suggests pattern recognition in AI review is not a stable rule, so relying on it for critical concurrency bugs is risky.