跪拜 Guibai
← Back to the summary

How the Claude Code Team Verifies the 90% of Code That Claude Writes

Anthropic's Addy Osmani published a post on claude.dev on November 16, introducing how the Claude Code team verifies code written by Claude. The team writes about 90% of its code with Claude and also reviews it with Claude. The human work shifts from reading every diff line to defining what "done" means and then checking evidence.

The first half of the article is the author's recommendations on test investment, recommending three types: end-to-end, property, and differential tests. The second half is the team's own practices, which can be grouped into four points:

Some of these have been built into Claude Code: the built-in /verify, the instruction to run the verify skill before committing since v2.1.286, and the adjustment to /code-review in v2.1.328 on November 19. The following combines the original practices with the corresponding product mechanisms.

The key of this approach is not adding more reviews, but making the checks trustworthy: checks are written by the code owner, placed on branches that changes cannot touch, and backtested with past incidents. The subtitle of the original post says just that: how to keep the verifier honest. It should be noted that the original only gives practices, not effect numbers.

Verification determines how many Claude instances you can run in parallel

The original judgment: without verification, a person can only run one or two Claude instances at a time because each output must be manually checked; with verification, you can run dozens or even hundreds in parallel.

Boris Cherny, the creator of Claude Code, also said in his October article on review processes that giving Claude a way to verify its work is the most important thing for getting good results. He estimates this can improve the final result by two to three times, which is his personal estimate.

The only quantified number in the article is "task success rate up to 10 percentage points". When quoting it, be clear about the source. It links to the Claude Managed Agents launch post from April, which is an internal test on structured file generation tasks: comparing a separate grading agent that checks results against a rubric to a normal prompt loop, with the biggest improvement on hard problems. This is not a number measured by the Claude Code team on its own codebase.

The cost of verification is also lower than before. Tests, benchmarks, specs, and end-to-end runs can now be generated by agents, and after generation they run as ordinary programs repeatedly without consuming much extra token.

If an agent repeatedly fails on the same type of check, the example in the original is "is there user data in the logs", add a hook or write an explicit rule in CLAUDE.md. The original says this basically avoids recurrence.

End-to-end, property, and differential tests

This section is the author's personal advice. Regardless of which test type you choose, first deal with two kinds of tests that can mislead agents:

End-to-end tests are the author's first choice. They describe real user flows, such as adding to cart, checkout, jumping to confirmation page. The functionality either works or not, and it's hard for an agent to cheat.

Unit tests are faster and more stable, but they can all pass while the feature is still broken because both the agent and these tests don't understand the full task. When an end-to-end test fails, it can give a screenshot and exactly what's broken, not just a timeout error.

The specific method is to use a skill to have the agent operate a real running system via CDP, simulators, or an API harness. A feature map compiled from the user's perspective, listing paths, shortcuts, selectors, and expected results, can serve as both an agent's operating guide and a scoring rubric.

Property testing specifically looks for edge cases that humans easily miss, and that's where agents often make mistakes. Write a statement you believe cannot be disproven, then generate thousands of inputs using fast-check (JS/TS), Hypothesis (Python), or proptest (Rust) to try to disprove it.

These inputs don't carry the bias of the agent that wrote the code. Property testing is especially useful in rewrite projects where the specification no longer matches production behavior and cannot be used as a verification baseline.

Differential testing is suitable for large-scale migrations where functionality should not change. The same set of inputs runs through both old and new systems; if results differ, it's a bug or a breaking change, and the old system is the answer key.

Inputs can be randomly generated data or samples of production traffic. Normalize irrelevant differences like whitespace before comparing; things that the new system should legitimately differ on go into an approved changes list.

The verify skill that runs before commit

The first practice of the team: verification is running the change, which is more important than running tests. The original uses the analogy: no one would release an iOS change without trying it in a simulator, and the same applies to Claude.

So the team has Claude test the entire app like a human or QA: web changes use Claude's Chrome extension, mobile uses iOS or Android MCP, desktop apps use computer use, and backend just runs the service locally. The original says almost all of Anthropic's changes are tested this way.

How to start simulators and run the app is written into a skill named verify, placed under .claude/skills/ in the repository, as close to the relevant code as possible, and as many as needed. Claude uses it before committing.

The original recommends investing effort to make this skill fast and stable, letting Claude iterate and stress-test it, because it's the project's smoke test.

In Claude Code, there are three corresponding mechanisms (docs read on November 20):

The original's phrase "Claude knows to use the verify skill before committing" corresponds to this commit instruction. The conditions for it to take effect are shown in the image above; the most common pitfall is the source: built-in /verify, skills in plugins, and skills in the claude.ai account do not count.

So to have Claude automatically verify before commit, the easiest way is to run /verify once in the project and commit the recorded SKILL.md into the repo. Claude checks this skill at session start, so it takes effect from the next session.

Three failed attempts to fix and the 39-item reliability checklist

No one can read all changes, so the team uses "failure" to decide which changes need human review. Claude runs checks and fixes failures itself; if the same check fails three consecutive times, Claude stops and brings in a human.

There's a similar cap in the October review article: after three rounds of review, the reviewer asks the author to step back and reconsider the approach.

Checks are written by the person responsible for that code and adjusted repeatedly until the issues they report match what that person would flag when reviewing code themselves. When a check needs to bring in a human, it must state why, so the brought-in person doesn't have to find the reason in the diff.

The team also compiled past incidents into a 39-item reliability checklist. Each item has three parts: a trigger condition that can be seen from the diff, a check, and the incident or PR that led to it.

One example from the original: "--resume can load old session records". If session storage or message types are changed, check that new fields can be loaded safely, because old session records don't have this field.

Claude reads the diff and only matches items whose trigger condition matches, typically 3 to 10 items. No one needs to remember all past incidents. The most worth copying is the format: trigger conditions written as facts discernible from the diff, so checks can be selected per change.

Review rules on a branch that changes cannot modify

The team doesn't improve review quality by stacking more reviewers. The original's judgment: adding a more expensive reviewer on top of the main one is often worse than tuning the main one. Tuning is done with a REVIEW.md that states what to focus on when reviewing this code.

Multiple review rounds and verification should be connected: each round carries context from previous rounds and past failures, rather than starting from scratch. The original says this is cheaper and finds more.

The most important rule in this section: if a change can modify the checks that judge it, those checks are invalid. This translates into three points:

The difference between these two methods is clearly explained in Claude Code docs:

What REVIEW.md writes and where

REVIEW.md is a Markdown file at the repo root, used only by the managed Code Review service (GitHub PR auto-review). There is no fixed format. It changes what the review reports, not the format of the review result.

Project-wide background still goes into CLAUDE.md. The local /code-review does not read REVIEW.md; rules that should apply to local review must be in CLAUDE.md.

Claude Code docs suggest writing based on problems encountered; common categories:

The October review article added the Claude Code team's own experience. REVIEW.md is written by the code owner and good ones are short: first state what "going wrong" looks like in this repo (docs repo: content wrong; config repo: route misconfigured); then limit Nits, list skipped paths, require file:line evidence; finally require the review summary to give the issue count first, and say "none" if there is nothing to fix. The team also separately checks whether the reviewer actually follows REVIEW.md.

Following these five categories, it looks roughly like this (example, replace rules with your own project's):

# Review Guide

## What counts as Important
Only issues that cause behavioral bugs, data leaks, or unrecoverable failures count as Important.
Naming, style, refactoring suggestions are at most Nit.

## Nit cap
No more than 3 Nits per review.

## Don't report
- Lint, formatting, type errors already checked by CI
- Generated code under src/generated/ and lock files

## Check every time
- If API return structure changes, update interface docs
- User-visible error messages must not contain internal error codes

## Evidence requirement
When judging that code behavior is problematic, provide file and line number, not just infer from function names.

## Summary format
First list how many issues were found; if none, say so directly.

Before writing, also confirm where it is and whether it will take effect:

Commit gate: record first, then block

The team stamps every commit in its repo, stating how the commit was verified. If not verified, write "none" so you can directly query which commits were unverified.

On top of that, the team built a commit gate: if the session hasn't run /code-review or /verify, block the commit. The gate has three modes: off, observe, enforce.

The section's title is "Start new gates in observe mode". The original doesn't say how the gate is implemented, nor the false-positive rate during observe.

Reviewer recall is calculated in three stages

The team regularly "reviews the reviewer", looking at three numbers daily: number of merged changes later reverted, number of changes rejected by maintainers before merge, and a control group of similarly sized changes that have had no problems.

Formal evaluation follows the Code Review team's approach from the October article:

Recall is computed in three stages: raw discovery, after verification, and final review. A single end-to-end score would hide whether the system filtered out already-found bugs in the middle. The team keeps increasing recall until further improvement is just overfitting to eval, not catching more bugs.

Beyond correctness, also consider whether changes are desired. Code Review labels each finding as Important, Nit, or Pre-existing; Pre-existing means the bug existed before and not the fault of this change's author.

On November 19, v2.1.328 adjusted local /code-review in this direction: on most models it only reports bugs and CLAUDE.md violations; cleanup like reuse, simplification, efficiency is handled by /simplify; each finding is labeled Important, Pre-existing, or Nit, with fix suggestions at the end of the summary.

Turn missed bugs into rules

In the Claude Code repo, missed issues become rules in two ways:

Flaky tests are also handled strictly: Claude cannot skip it, cannot add sleep, cannot retry until it passes. If root cause isn't found, delete the test and write the reason.

Rules are also cleaned up regularly. Test sets, verification steps, and various .md files accumulate if left alone.

Original data boundaries and order of adoption

The original gives practices, not effects. The 39-item checklist, verify skill, REVIEW.md, and commit gate have no numbers like failure rate or false-positive rate; the commit gate's implementation isn't written either. The two numbers in the article, "up to 10 percentage points" from Managed Agents' structured file generation test, and "two to three times" as Boris Cherny's personal estimate.

When adopting, sort by increasing investment, using only practices from the original and docs:

  1. First get Claude to run the app. Fix too-slow and flaky tests; run /verify once in the project and commit the recorded .claude/skills/verify/SKILL.md; from v2.1.286 Claude runs it before every commit; then use a Stop hook or /goal to ensure tests, build, and linter run before PR.
  2. Put rules where changes can't touch them. Local review rules go into CLAUDE.md; for managed Code Review, put REVIEW.md on the default branch and protect it. Set the three-failed-attempts handoff; accumulate a reliability checklist from your own incidents in the format "diff trigger, check, origin".
  3. Finally add the commit gate and reviewer evaluation. First stamp commits with verification status, including none; run the gate in observe for a while, see false positives, then switch to enforce; keep the escape hatch of passing after two blocks; then do staged recall evaluation and periodic rule cleanup.

The original concludes: with these in place, the human job shifts from reading every diff to defining what "done" means and checking evidence, which is also the prerequisite for running many Claude instances in parallel.

Reference links