95% Code Coverage, 3,000 Test Cases, and a Production Outage on Day One

The thing is this.

Some time ago, a test lead came to me and said his team spent two months writing over 3,000 test cases and pushed code coverage to 95%. The boss was very satisfied, and the team felt confident.

The day after going live, it blew up.

An extremely serious bug surfaced online, and none of the 3,000 cases caught it. It wasn't some extreme edge-case scenario; it was a very ordinary operation path that the test team simply hadn't thought of.

3,000 cases, and not a single one hit it.

He was especially frustrated. They had done everything they were supposed to: the case count was high enough, the coverage was high enough, but when it came to production, what was supposed to leak still leaked.

After listening, I had only one thing to say: you tested a lot, but you didn't test the right things.

This is too common.

I've seen too many teams equate test quality with test quantity. They write thousands of cases, push coverage above 90%, and feel confident.

But coverage measures whether your code has been executed, not whether it executed correctly.

You write 100 cases that run every line of code, and coverage is 100%. But if those 100 cases are all ideal inputs and the smoothest paths, you've only repeatedly confirmed one thing: the code runs under normal conditions. That's it.

The places where things actually go wrong — abnormal inputs, concurrency conflicts, boundary conditions, module integration — not a single one was tested.

Coverage is a good metric, but don't deify it

Let me be clear first: I'm not denying coverage.

Coverage is a useful reference. It tells you which code hasn't been tested at all, which is a valuable signal. If a core module has only 20% coverage, then it indeed needs supplementing.

But too many teams treat coverage as a goal, not a reference.

There's Goodhart's Law in economics. Simply put, when a measure becomes a target, it ceases to be a good measure.

It's exactly the same in testing.

Once you set coverage as a KPI, team behavior distorts. To push coverage from 85% to 90%, people start writing a flood of cases that just run through without substantive assertions. A test function calls a target function, doesn't check the return value, doesn't check side effects — it contributes to coverage, but its testing value is roughly zero.

The numbers look good, but the quality hasn't changed.

INRIA conducted a large-scale study analyzing numerous open-source projects, and the conclusion was direct: at the project level, the correlation between coverage and the number of bugs found post-release is so weak it's negligible.

You read that right. Projects with high coverage don't necessarily have fewer bugs.

There's an even more insidious problem. Coverage doesn't distinguish the importance of code. Testing a logging utility class to 100% and testing core payment logic to 100% carry the same weight in a coverage report. But a bug in the former might at most mess up a log format, while a bug in the latter is a financial incident.

Focusing on coverage consumes a lot of energy on low-risk code, leaving no time to properly test the truly high-risk areas.

Testing accurately beats testing a lot

So what defines good testing?

In one sentence: test what should be tested.

Good testing isn't about covering everything; it's about precisely hitting high-risk areas. You don't need to test every line of code; you need to test the paths where the consequences of failure are most severe.

In testing methodology, this is called risk-driven testing. The idea is simple: prioritize by "probability of a bug" multiplied by "consequence of a bug." High probability and severe consequences? Test heavily. Low probability and light consequences? Spend less time or don't test at all.

Let's take a concrete example.

In an e-commerce system, can you treat the payment chain and a log cleanup script the same way? A bug in the payment chain means a user pays but doesn't receive the goods — complaints, refunds, trust collapse.

A bug in the log cleanup script at most fills up the disk and triggers an alert. For the former, you should spend significant effort writing boundary tests, exception tests, and concurrency tests; for the latter, a basic smoke test is enough.

But if you're fixated on coverage, these two pieces of code might get the same attention.

That's the problem.

Testing resources are always finite. You can't test everything; what you need to do is pour your limited resources into the most valuable places.

How to find what should be tested

The principles are laid out; now comes the practical question. How do you know which places should be tested?

From my own experience, look along these lines.

First, watch real failures.

Bugs that actually occurred in production are the highest-quality source of test requirements. Every production incident should result in a test case. Not vaguely saying "be more careful next time," but actually writing a case that locks down that scenario, ensuring the same bug won't reappear.

Many teams write great post-mortems and list a bunch of improvement measures, but they never materialize into automated tests. Two months later, they step into the same hole in a different posture.

Real failures have another benefit. They tell you that places you thought were safe are actually unsafe. You test core functionality the most, but what blows up online is often that corner you thought was "so simple it couldn't possibly go wrong." Real data slaps you in the face, and after the slap, you know where to patch.

Second, watch frequently changed areas.

Code change frequency is a great risk signal. If a module has been changed 20 times in three months, either the requirements are constantly shifting, or the logic itself is complex and unstable. Either way, it's a bug hotspot.

Manually running through it after each change and thinking it's fine to commit will eventually backfire. For frequently changed modules, regression testing must be automated, and assertions must be detailed.

Third, watch integration points.

A single module runs fine on its own, but problems arise when it's plugged into the system. This happens every day.

Interface contracts between services, data format passing, the timing of asynchronous messages, database transaction isolation levels — bugs at these integration points are far more numerous than bugs in unit logic. Yet these are precisely the places many people are too lazy to test, because setting up the environment is troublesome, creating data is troublesome, and running them is slow.

If you avoid testing because it's troublesome, once it goes live, the users will test it for you.

Fourth, validate with mutation testing.

The first three help you find "what places to test." Mutation testing solves another problem: whether the tests you wrote are actually useful.

The principle is straightforward. Actively inject some bugs into your code — like changing a greater-than sign to a less-than sign, a plus to a minus, deleting a line of validation logic — and then run your test suite. If the tests fail, it means your tests can catch this bug; they are effective. If all tests pass, it means your tests didn't check this logic at all; they were written for nothing.

This is much more honest than looking at coverage. Coverage tells you the code was executed; mutation testing tells you whether it was actually checked after execution.

For mainstream mutation testing tools, Java has PIT, Python has mutmut, JS has Stryker. Integrate them into CI and run them periodically. You'll be surprised to find that a test suite with 90% coverage might only have a mutation score of 40% to 50%. This means more than half of your tests are pretending to protect you.

In AI Agent scenarios, precision testing is even more critical

What we've discussed so far are mainly ideas for traditional software testing. If you're working on AI Agents, this matter is even more extreme.

With traditional software, you can at least calculate coverage; the input space, while large, is at least finite. An Agent's input space is infinite. Users can ask any question, operate in any order, and provide data in any format. Exhaustive enumeration is impossible.

More troublesome is the Agent's non-determinism. The same input might output A today and B tomorrow. A case passing once doesn't mean it passes stably. Running it 10 times might pass 7 and fail 3. Judging by the result of one run is no different from flipping a coin.

In this situation, precision testing isn't an optimization; it's a survival requirement.

How to choose? The same logic as before.

Extract failure cases from real user conversations. The types of problems users complain about most are precisely the scenarios where you should build evaluators. Spending a week synthesizing 1,000 test data points is far less valuable than pulling 50 real failure cases from production; the latter is ten times more valuable.

Watch core business chains. If your Agent is for customer service, the return/refund process is a hundred times more important than small-talk scenarios. Testing the entire return/refund chain thoroughly is more effective than exhaustively testing a hundred edge scenarios.

Run multiple times to check probability. Run a case at least 5 to 10 times and look at the pass rate. Passing once isn't passing; passing eight or more times out of ten is reliable.

A few practical experiences

We've covered the principles and directions. Finally, here are a few things I've summarized after stepping into pitfalls myself.

First, periodically clean up low-value tests.

More tests aren't always better. Cases that just run through, have vague assertions, and have never caught a bug are liabilities, not assets. They slow down CI, increase maintenance costs, and give you a false sense of security. Review them periodically; delete what should be deleted, refactor what should be refactored. A good test suite is refined, not bloated.

Second, track escape defects, not coverage.

Escape defects are bugs found in production. This number is much more honest than coverage. If coverage is 95% but every release has bugs surfacing online, your testing system has a problem. If coverage is 70% but production is almost incident-free, it means that 70% was all spent on the cutting edge.

The metric you should really be watching is: how many things you tested still blew up in production.

Third, testing follows business risk, not code volume.

Places with more code don't necessarily need more testing; places with higher risk do. Before every new feature goes live, do a risk assessment first. Which paths involve money, data security, or key nodes of user experience? Test those places thoroughly first, then consider the rest.

Fourth, for Agent evaluation, quality is far more important than quantity.

50 evaluation cases extracted from real business scenarios are far more useful than 500 synthetic data points. One case that can catch a core problem is worth more than a hundred waste cases running the happy path. The value of evaluation doesn't lie in how many data points you have, but in whether your data reflects real business quality.

Final words

Let's go back to that test lead at the beginning.

3,000 cases, 95% coverage — sounds impressive. But if those 3,000 cases were all testing whether code was executed, without hitting the real risk points, then those 3,000 cases are just a pile of pretty numbers.

The value of testing has never been in quantity. It's in precision.

Testing 100 places is not as good as accurately testing the 10 most critical ones. Writing 1,000 cases is not as good as having 100 that can truly catch bugs.

Good testing isn't about testing a lot. It's about testing accurately.

Knowing where to test, where not to test, and where you must test to the death. This judgment is a hundred times more important than the ability to write cases.