Testing · A/B Testing · Interview

95% Code Coverage, 3,000 Test Cases, and a Production Outage on Day One

By 狂师 · Aug 18, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Coverage-driven testing culture is widespread and expensive: it consumes CI time, creates maintenance drag, and delivers false confidence while real failure modes go unexamined. Teams that switch to risk-driven testing and mutation testing catch more bugs with fewer cases and stop treating coverage dashboards as safety certificates.

Summary

A team spent two months writing over 3,000 test cases and hitting 95% code coverage, only to have a serious bug surface in production the day after launch. The failure wasn't an exotic edge case — it was a common user path the suite never considered. The incident illustrates how coverage measures code execution, not correctness, and how treating coverage as a KPI incentivizes hollow tests that inflate metrics without catching bugs.

Risk-driven testing flips the approach: prioritize paths by the probability of failure multiplied by its business impact. Payment logic demands exhaustive boundary, concurrency, and exception testing; a log-cleanup script needs only a smoke test. Real production failures, frequently changed modules, and service integration points are the highest-signal targets for test effort. Mutation testing tools like PIT, mutmut, and Stryker reveal how many high-coverage tests are actually checking nothing.

For AI Agent systems, precision testing becomes existential. Infinite input spaces and non-deterministic outputs make exhaustive coverage impossible. Pulling 50 real failure cases from user conversations is worth more than 1,000 synthetic data points. The core discipline is the same: track escape defects, not coverage percentages, and let business risk — not code volume — decide where testing effort goes.

Takeaways

— Code coverage measures whether lines were executed, not whether they executed correctly; a 100%-coverage suite of happy-path cases confirms nothing about real-world correctness.

— Setting coverage as a KPI triggers Goodhart's Law: teams write assertion-free tests that inflate the number without adding protection.

— An INRIA study across open-source projects found negligible correlation between coverage percentage and post-release bug counts.

— Risk-driven testing ranks test targets by (probability of bug) × (business consequence); payment logic gets exhaustive testing, log utilities get a smoke test.

— Production incidents are the highest-quality source of new test cases — every escape defect should produce an automated regression test that locks down the exact scenario.

— Frequently changed modules are bug hotspots and need automated regression suites with detailed assertions.

— Integration points — service contracts, data formats, async message timing, transaction isolation — produce more bugs than unit logic but are often skipped because they're harder to set up.

— Mutation testing (PIT for Java, mutmut for Python, Stryker for JS) injects deliberate bugs to verify that tests actually detect them; suites with 90% coverage often score only 40–50% on mutation tests.

— AI Agent testing cannot rely on coverage at all — infinite input space and non-deterministic outputs make precision testing a survival requirement, not an optimization.

— 50 real failure cases extracted from production user conversations are worth far more than 500 synthetic test data points for Agent evaluation.

— Run Agent test cases at least 5–10 times and measure pass rate; a single pass proves nothing when outputs are non-deterministic.

— Periodically delete low-value tests that have never caught a bug — they slow CI, increase maintenance, and create false confidence.

— Track escape defects (bugs found in production) as the primary quality metric, not coverage; 70% coverage with zero escapes beats 95% coverage with recurring incidents.

Conclusions

Coverage percentage and bug count are so weakly correlated that treating coverage as a quality gate is statistical noise dressed as engineering rigor.

Mutation testing exposes an uncomfortable truth most teams avoid: a large fraction of automated tests are performative, checking nothing of consequence while burning CI minutes.

The gap between what teams test (happy paths in isolated units) and what breaks in production (integration surprises, concurrency races, mundane user paths nobody thought to script) is a structural blind spot in coverage-driven cultures.

AI Agent testing inverts the traditional testing economics — synthetic data generation is nearly worthless compared to curating real failure cases, and statistical pass rates replace binary pass/fail assertions.

Test-case count as a pride metric is a liability signal; a lean suite that catches escape defects is worth more than a bloated suite that only confirms the code runs under ideal conditions.

Concepts & terms

Goodhart's Law

An adage stating that when a measure becomes a target, it ceases to be a good measure. In testing, making code coverage a KPI causes teams to write hollow tests that boost the metric without improving quality.

Risk-driven testing

A testing strategy that prioritizes test effort by multiplying the probability of a bug by the severity of its business consequence, rather than trying to test everything equally.

Mutation testing

A technique that deliberately injects faults (mutations) into source code and runs the test suite to see if any tests fail. A low mutation score reveals tests that execute code without actually verifying its behavior.

Escape defect

A bug discovered in production that was not caught by any pre-release testing phase. Tracking escape defects measures how well the test suite protects against real failures.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗