AI Reviewers Can't Be Bribed into Leniency: A 5-Bug Experiment with Four Personas
AI code review catches planted defects reliably, but severity ratings shift from P0 to P2 across runs, so a single model's triage call is a lottery. Teams can use AI as a detection filter, then apply voting across multiple reviewers or human review for prioritization.
kyriewen set out to test the urban legend that AI-written code reviewed by AI is just plagiarists grading each other. They wrote a 50-line React search hook with five planted defects: an uncleaned timer causing race conditions, a never-reset loading flag, unescaped user input in RegExp, an innerHTML XSS, and a missing dependency. Four AI reviewers, each with a distinct persona—senior neutral, harsh nitpicker, security-focused, and a deliberately encouraging 'team vibe' reviewer—were asked to review. The 'nice' reviewer opened with praise and a tilde, then flagged all five issues without mercy.
The author then tried three tricks to force leniency: inserting the overly kind persona, hand-crafting a textbook water-down review (LGTM with nitpicks), and telling the AI that the code was written by a human newbie versus an AI. Every approach failed. The forged review was rejected by three fresh AI instances, which re-found all five bugs. The human-newbie attribution got a polite Block, the AI attribution got a stark stereotype-driven Block.
The most useful finding came from severity inconsistency: the same bug got P0, P1, and P2 from different instances. The author concludes that detection is reliable, but severity ranking should be decided by voting across multiple reviewers, not by trusting a single model.
The persona prompt alters how an AI reviewer phrases feedback but not how thoroughly it checks, so softer wording should never be read as softer judgment.
Inconsistent severity levels for the same bug across instances mean an AI reviewer can serve as a detector but not as a reliable judge of impact.
The experiment's clean results on a 50-line snippet don't transfer automatically to a sprawling legacy module, where context loss and ambiguity would likely degrade recall.