How the Claude Code Team Verifies the 90% of Code That Claude Writes
Anthropic's Addy Osmani published a post on claude.dev on November 16, introducing how the Claude Code team verifies code written by Claude. The team writes about 90% of its code with Claude and also reviews it with Claude. The human work shifts from reading every diff line to defining what "done" means and then checking evidence.
The first half of the article is the author's recommendations on test investment, recommending three types: end-to-end, property, and differential tests. The second half is the team's own practices, which can be grouped into four points:
- Changes must actually run the app for verification. The steps to start and operate the app are written into a skill named verify, which Claude runs before committing.
- If a check fails, Claude fixes it itself; only after three consecutive failed attempts does it hand over to a human. Which risks to check is determined by a 39-item reliability checklist compiled from past incidents, and each time only the items relevant to the current change are checked.
- The rules for judging changes are placed on a branch that the changes cannot modify, and the review result is only valid for one commit.
- The reviewer itself is also evaluated, and missed bugs are turned into new rules.
Some of these have been built into Claude Code: the built-in /verify, the instruction to run the verify skill before committing since v2.1.286, and the adjustment to /code-review in v2.1.328 on November 19. The following combines the original practices with the corresponding product mechanisms.
The key of this approach is not adding more reviews, but making the checks trustworthy: checks are written by the code owner, placed on branches that changes cannot touch, and backtested with past incidents. The subtitle of the original post says just that: how to keep the verifier honest. It should be noted that the original only gives practices, not effect numbers.
Verification determines how many Claude instances you can run in parallel
The original judgment: without verification, a person can only run one or two Claude instances at a time because each output must be manually checked; with verification, you can run dozens or even hundreds in parallel.
Boris Cherny, the creator of Claude Code, also said in his October article on review processes that giving Claude a way to verify its work is the most important thing for getting good results. He estimates this can improve the final result by two to three times, which is his personal estimate.
The only quantified number in the article is "task success rate up to 10 percentage points". When quoting it, be clear about the source. It links to the Claude Managed Agents launch post from April, which is an internal test on structured file generation tasks: comparing a separate grading agent that checks results against a rubric to a normal prompt loop, with the biggest improvement on hard problems. This is not a number measured by the Claude Code team on its own codebase.
The cost of verification is also lower than before. Tests, benchmarks, specs, and end-to-end runs can now be generated by agents, and after generation they run as ordinary programs repeatedly without consuming much extra token.
If an agent repeatedly fails on the same type of check, the example in the original is "is there user data in the logs", add a hook or write an explicit rule in CLAUDE.md. The original says this basically avoids recurrence.
End-to-end, property, and differential tests
This section is the author's personal advice. Regardless of which test type you choose, first deal with two kinds of tests that can mislead agents:
- Flaky tests cause agents to learn to rerun for luck rather than fix code.
- Too-slow tests cause agents to run fewer times per batch of changes, making problems harder to locate. Heavy tests should run before commit.
End-to-end tests are the author's first choice. They describe real user flows, such as adding to cart, checkout, jumping to confirmation page. The functionality either works or not, and it's hard for an agent to cheat.
Unit tests are faster and more stable, but they can all pass while the feature is still broken because both the agent and these tests don't understand the full task. When an end-to-end test fails, it can give a screenshot and exactly what's broken, not just a timeout error.
The specific method is to use a skill to have the agent operate a real running system via CDP, simulators, or an API harness. A feature map compiled from the user's perspective, listing paths, shortcuts, selectors, and expected results, can serve as both an agent's operating guide and a scoring rubric.
Property testing specifically looks for edge cases that humans easily miss, and that's where agents often make mistakes. Write a statement you believe cannot be disproven, then generate thousands of inputs using fast-check (JS/TS), Hypothesis (Python), or proptest (Rust) to try to disprove it.
These inputs don't carry the bias of the agent that wrote the code. Property testing is especially useful in rewrite projects where the specification no longer matches production behavior and cannot be used as a verification baseline.
Differential testing is suitable for large-scale migrations where functionality should not change. The same set of inputs runs through both old and new systems; if results differ, it's a bug or a breaking change, and the old system is the answer key.
Inputs can be randomly generated data or samples of production traffic. Normalize irrelevant differences like whitespace before comparing; things that the new system should legitimately differ on go into an approved changes list.
The verify skill that runs before commit
The first practice of the team: verification is running the change, which is more important than running tests. The original uses the analogy: no one would release an iOS change without trying it in a simulator, and the same applies to Claude.
So the team has Claude test the entire app like a human or QA: web changes use Claude's Chrome extension, mobile uses iOS or Android MCP, desktop apps use computer use, and backend just runs the service locally. The original says almost all of Anthropic's changes are tested this way.
How to start simulators and run the app is written into a skill named verify, placed under .claude/skills/ in the repository, as close to the relevant code as possible, and as many as needed. Claude uses it before committing.
The original recommends investing effort to make this skill fast and stable, letting Claude iterate and stress-test it, because it's the project's smoke test.
In Claude Code, there are three corresponding mechanisms (docs read on November 20):
- Built-in /verify: no configuration needed, infers how to start based on project type and README, package.json, Makefile, builds and runs the app to confirm the change works, does not fall back to tests or type checking. When the project requires a database, env files, a GUI, or multi-step build, inference becomes unreliable; in that case, run /run-skill-generator first, which starts the app from a clean environment and records the working steps as a project skill.
- Successful steps are recorded: when /verify doesn't have existing steps, it writes the working steps to
.claude/skills/verify/SKILL.mdin the repo root (or the changed package directory in a monorepo), and this file then replaces the built-in /verify. Claude only modifies this file if it leads a run astray, so it can be committed directly. - Runs before commit: starting from v2.1.286, if a skill named verify or simplify exists at session start, Claude Code's built-in commit instruction requires Claude to run it before each commit, except for commits that only change docs or tests.
The original's phrase "Claude knows to use the verify skill before committing" corresponds to this commit instruction. The conditions for it to take effect are shown in the image above; the most common pitfall is the source: built-in /verify, skills in plugins, and skills in the claude.ai account do not count.
So to have Claude automatically verify before commit, the easiest way is to run /verify once in the project and commit the recorded SKILL.md into the repo. Claude checks this skill at session start, so it takes effect from the next session.
Three failed attempts to fix and the 39-item reliability checklist
No one can read all changes, so the team uses "failure" to decide which changes need human review. Claude runs checks and fixes failures itself; if the same check fails three consecutive times, Claude stops and brings in a human.
There's a similar cap in the October review article: after three rounds of review, the reviewer asks the author to step back and reconsider the approach.
Checks are written by the person responsible for that code and adjusted repeatedly until the issues they report match what that person would flag when reviewing code themselves. When a check needs to bring in a human, it must state why, so the brought-in person doesn't have to find the reason in the diff.
The team also compiled past incidents into a 39-item reliability checklist. Each item has three parts: a trigger condition that can be seen from the diff, a check, and the incident or PR that led to it.
One example from the original: "--resume can load old session records". If session storage or message types are changed, check that new fields can be loaded safely, because old session records don't have this field.
Claude reads the diff and only matches items whose trigger condition matches, typically 3 to 10 items. No one needs to remember all past incidents. The most worth copying is the format: trigger conditions written as facts discernible from the diff, so checks can be selected per change.
Review rules on a branch that changes cannot modify
The team doesn't improve review quality by stacking more reviewers. The original's judgment: adding a more expensive reviewer on top of the main one is often worse than tuning the main one. Tuning is done with a REVIEW.md that states what to focus on when reviewing this code.
Multiple review rounds and verification should be connected: each round carries context from previous rounds and past failures, rather than starting from scratch. The original says this is cheaper and finds more.
The most important rule in this section: if a change can modify the checks that judge it, those checks are invalid. This translates into three points:
- Rules are read from a branch that changes cannot modify. The managed Code Review service only reads REVIEW.md from the PR's base branch or default branch, never from the PR's own changes; to change review behavior, you must first merge the REVIEW.md change. The docs also recommend protecting the default branch. The team's rules for intercepting changes are also placed on branches that the changes cannot modify. The local /code-review command reads CLAUDE.md, not REVIEW.md.
- Review results are only valid for one commit. If someone pushes a new commit during the review, the review is invalidated. The original says: "A passing review on an out-of-date commit means nothing."
- Cheap checks run first. An agent that pushes without local tests spends shared review and CI time. The team has Claude run tests, build, and linter before opening a PR. Two enforcement methods: a hook that runs tests when Claude finishes, or a /goal with test completion as the condition.
The difference between these two methods is clearly explained in Claude Code docs:
- Stop hook is written in settings, applies to all sessions in scope, triggers at the end of each Claude reply, and can run scripts for deterministic checks. If the hook returns
"decision": "block"or exits with code 2, Claude continues working. There's a cap on consecutive continuations: after 8 consecutive blocks, the 9th block is ignored and the turn ends; a tool call resets the counter; the cap can be raised with environment variableCLAUDE_CODE_STOP_HOOK_BLOCK_CAP. - /goal is a built-in Stop hook that only applies to the current session. After setting a completion condition, a model checks at the end of each turn whether the condition is met; if not, the next turn begins.
What REVIEW.md writes and where
REVIEW.md is a Markdown file at the repo root, used only by the managed Code Review service (GitHub PR auto-review). There is no fixed format. It changes what the review reports, not the format of the review result.
Project-wide background still goes into CLAUDE.md. The local /code-review does not read REVIEW.md; rules that should apply to local review must be in CLAUDE.md.
Claude Code docs suggest writing based on problems encountered; common categories:
- Important mislabeled: redefine what counts as Important vs. at most Nit in this repo. The default standard is set for production code; docs repos, config repos, or prototypes may need a much narrower definition; you can also upgrade, e.g., any violation of CLAUDE.md counts as Important.
- Too many Nits: limit the number of Nits per review, the doc example is 5.
- Commenting on generated files or things CI already checks: list paths and problem categories to skip, like generated code, lock files, vendored dependencies, and lint/spelling that CI already checks. For directories that deserve light review, raise the threshold, e.g., only report almost-certain and severe issues in
scripts/. - Team rules nobody checks: write rules to check on every PR, e.g., new API routes must have integration tests.
- A type of finding often misreported: require evidence before reporting, e.g., claims about code behavior must cite
file:line, not just infer from naming.
The October review article added the Claude Code team's own experience. REVIEW.md is written by the code owner and good ones are short: first state what "going wrong" looks like in this repo (docs repo: content wrong; config repo: route misconfigured); then limit Nits, list skipped paths, require file:line evidence; finally require the review summary to give the issue count first, and say "none" if there is nothing to fix. The team also separately checks whether the reviewer actually follows REVIEW.md.
Following these five categories, it looks roughly like this (example, replace rules with your own project's):
# Review Guide
## What counts as Important
Only issues that cause behavioral bugs, data leaks, or unrecoverable failures count as Important.
Naming, style, refactoring suggestions are at most Nit.
## Nit cap
No more than 3 Nits per review.
## Don't report
- Lint, formatting, type errors already checked by CI
- Generated code under src/generated/ and lock files
## Check every time
- If API return structure changes, update interface docs
- User-visible error messages must not contain internal error codes
## Evidence requirement
When judging that code behavior is problematic, provide file and line number, not just infer from function names.
## Summary format
First list how many issues were found; if none, say so directly.
Before writing, also confirm where it is and whether it will take effect:
- Review reads REVIEW.md on the PR's base branch or default branch; a change to it in the PR takes effect only after merging.
- The three instructions (lower severity, exclude paths, limit Nits) are only executed when the default branch is protected.
- When the default branch is protected, REVIEW.md in subdirectories can add checks for changes under that directory; those in tests, examples, generated, vendor do not take effect.
- If rules get too many, split into several files and include them with a single line
@filenamein the root REVIEW.md; included files are in the root, max 8, total under 150,000 bytes; you can also use@.claude/skills/<directory>/SKILL.mdto include a skill's content. Inclusion also only works on protected branches.
Commit gate: record first, then block
The team stamps every commit in its repo, stating how the commit was verified. If not verified, write "none" so you can directly query which commits were unverified.
On top of that, the team built a commit gate: if the session hasn't run /code-review or /verify, block the commit. The gate has three modes: off, observe, enforce.
- observe only records commits that would have been blocked, doesn't actually block, used to review false positives, affects no one.
- enforce actually blocks, but you can write the reason for no verification in the commit message. If blocked twice in the same session, the next one passes, to avoid Claude getting stuck, because Claude Code isn't perfect either.
The section's title is "Start new gates in observe mode". The original doesn't say how the gate is implemented, nor the false-positive rate during observe.
Reviewer recall is calculated in three stages
The team regularly "reviews the reviewer", looking at three numbers daily: number of merged changes later reverted, number of changes rejected by maintainers before merge, and a control group of similarly sized changes that have had no problems.
Formal evaluation follows the Code Review team's approach from the October article:
- Samples are taken from past incidents and reverts. Each sample restores the repo to the state at review time, with one sentence describing the bug; only findings that accurately describe the bug score.
- Use merged and stable changes as a control to prevent over-reporting.
- Sample sets for tuning are separate from the fixed test set for evaluation.
Recall is computed in three stages: raw discovery, after verification, and final review. A single end-to-end score would hide whether the system filtered out already-found bugs in the middle. The team keeps increasing recall until further improvement is just overfitting to eval, not catching more bugs.
Beyond correctness, also consider whether changes are desired. Code Review labels each finding as Important, Nit, or Pre-existing; Pre-existing means the bug existed before and not the fault of this change's author.
On November 19, v2.1.328 adjusted local /code-review in this direction: on most models it only reports bugs and CLAUDE.md violations; cleanup like reuse, simplification, efficiency is handled by /simplify; each finding is labeled Important, Pre-existing, or Nit, with fix suggestions at the end of the summary.
Turn missed bugs into rules
In the Claude Code repo, missed issues become rules in two ways:
- Look for patterns from human review comments and PR reverts. A pattern that appears in at least two PRs and is identifiable from the diff becomes a candidate rule; then have three agents try to disprove it, and finally a human approves it in a PR.
- Replay PRs associated with known incidents, with three bug-finding agents. If at least two find it, it's a detectable bug; if it's missed even on a small diff, it's worth writing into REVIEW.md.
Flaky tests are also handled strictly: Claude cannot skip it, cannot add sleep, cannot retry until it passes. If root cause isn't found, delete the test and write the reason.
Rules are also cleaned up regularly. Test sets, verification steps, and various .md files accumulate if left alone.
Original data boundaries and order of adoption
The original gives practices, not effects. The 39-item checklist, verify skill, REVIEW.md, and commit gate have no numbers like failure rate or false-positive rate; the commit gate's implementation isn't written either. The two numbers in the article, "up to 10 percentage points" from Managed Agents' structured file generation test, and "two to three times" as Boris Cherny's personal estimate.
When adopting, sort by increasing investment, using only practices from the original and docs:
- First get Claude to run the app. Fix too-slow and flaky tests; run /verify once in the project and commit the recorded
.claude/skills/verify/SKILL.md; from v2.1.286 Claude runs it before every commit; then use a Stop hook or /goal to ensure tests, build, and linter run before PR. - Put rules where changes can't touch them. Local review rules go into CLAUDE.md; for managed Code Review, put REVIEW.md on the default branch and protect it. Set the three-failed-attempts handoff; accumulate a reliability checklist from your own incidents in the format "diff trigger, check, origin".
- Finally add the commit gate and reviewer evaluation. First stamp commits with verification status, including none; run the gate in observe for a while, see false positives, then switch to enforce; keep the escape hatch of passing after two blocks; then do staged recall evaluation and periodic rule cleanup.
The original concludes: with these in place, the human job shifts from reading every diff to defining what "done" means and checking evidence, which is also the prerequisite for running many Claude instances in parallel.
Reference links
- How the Claude Code team thinks about verification (2026-11-16): https://claude.dev/blog/how-the-claude-code-team-thinks-about-verification/
- How the Claude Code team does code review (2026-10-15): https://claude.dev/blog/solving-code-review-at-anthropic/
- Claude Managed Agents launch post (2026-04-08): https://claude.com/blog/claude-managed-agents
- Building verification loops in Claude Code with skills (2026-07-22): https://claude.com/blog/building-verification-loops-in-claude-code-with-skills
- Claude Code docs Skills (Run and verify your app): https://code.claude.com/docs/en/skills
- Claude Code docs Customize Code Review (REVIEW.md): https://code.claude.com/docs/en/code-review-customize
- Claude Code docs Hooks (Stop): https://code.claude.com/docs/en/hooks
- Claude Code docs /goal: https://code.claude.com/docs/en/goal
- Claude Code CHANGELOG: https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md