Six Ways a Production Agent Dies: A 200K-User Project's Four Hard-Won Refactors
The 6 Ways an Agent Dies After Launch: A 200K-User Project's Four Rewrites
We built a business-facing Agent for end users, serving 200,000+ users with 500,000+ daily conversations. We rewrote it four times, killed about 90% of hallucinations, and dropped P95 from 15s+ to under 10s. This post doesn't brag about results. It tells you how each version screwed itself, and what we understood after each death.
0. First, the conclusion: none of the four versions were designed
If you are shipping an Agent, remember this: none of the four versions was "designed." Every one was pushed forward by the problems the previous version exposed.
We were solving the same set of contradictions the entire time:
- Hallucinations — the model makes things up
- Token cost — we keep stuffing more context in
- Response latency — more reasoning turns means slower
- Operability — change one thing, and the whole business needs re-review
These four pull against each other: less hallucination demands more constraints and more recall, so tokens go up and latency goes up; faster means less reasoning, so hallucinations go up; easier maintenance means splitting prompts, but then the segments fight each other.
You can tell whether an Agent team is mature by whether they can articulate where the contradiction is in this version. If they can't, they're probably still on version 1.
Figure 1 · Four versions of evolution: each one was pushed by the previous one
Death #1: Heavy reliance on intent recognition — every new business meant retraining the algorithm
The start was small
In November 2024, V1 launched.
The origin was plain: we pulled the top 10 daily customer-service questions and found a huge number of "why was my order flagged as abnormal." Users didn't know the basis for the judgment and kept asking.
The intention was just to turn my own repetitive work into an SOP for users — not to build a general-purpose assistant.
Architecture
A pre-stage intent recognition layer routed to the corresponding tool or subagent based on static intent.
user query → intent recognition → routing → tool/subagent → reply
It did one thing right
It got the Agent's front-end/back-end SSE streaming pipeline working.
That value is seriously underrated. Streaming is the foundation of the whole experience — users see characters appear one by one while waiting, rather than staring at a spinner for 15 seconds. Many teams patch this in last, and then have to redo the entire timeout budget for the chain.
It died on three things
- No multi-turn capability — the second question lost the context
- Heavy dependence on intent — if the intent was misclassified, everything downstream was wrong, and confidently so
- Every business integration required the algorithm to retrain parameters — delivery cycles measured in weeks
The third was fatal. Every new scenario meant scheduling algorithm time, labeling data, training a model, and evaluating results. The business side couldn't wait, and we couldn't sustain it.
Why we later dropped the traditional classification model
Because that's what V1 did.
The problem with a classification model isn't accuracy; it's that it can only say "which category," not "which specific order. " It can't handle "this order," "the last one," or "the one I mentioned just now." And every new scenario required retraining the algorithm.
The only benefit is that it can output observation metrics per intent, which helps the business collect revenue data. In other words, a classification model gives you observability, not capability.
Death #2: Stuff everything into the LLM — tokens went from 3,200 to 12,000
Why we rewrote
In 2025, models got noticeably stronger. We saw that the assistant was very stupid: it read but didn't understand, gave the same answer for multi-turn inputs.
We decided to rewrite. Core idea: don't rely heavily on intent recognition, let the LLM handle the whole reasoning and decision-making, and support multi-turn.
Architecture: looks perfectly standard
Gateway layer → auth, security checks, save conversation
Context layer → user info, ticket info, anomaly info
Orchestration layer → RAG retrieval
Capability layer → tool, skill
core agent → ReAct loop reasoning
Model layer → cloud vendor model service / internal agent platform
+ monitoring and evaluation
This architecture would pass scrutiny in any Agent architecture article.
It died on one sentence
We dumped all tools and all context into the core agent in one go.
The result was inevitable:
- Tokens exploded from 3,200 to 12,000
- Hallucinations mounted
- Rules were violated
- Responses timed out
But the most fatal part wasn't these
It was that nobody dared touch the prompt.
All business shared one prompt. Change one part, and every business had to re-review before shipping.
One prompt became the most feared file in the company. Change one character, go through three rounds of review, pull in five business stakeholders. In the end, people preferred patching the code instead of the prompt — because a patch at least didn't affect everyone else.
Root cause
One sentence: we mixed "choosing capabilities" with "executing capabilities."
Having one model both decide "which tool to use" and "how to use the tool to complete the task" made it flip-flop between the two goals. The larger the candidate pool, the more it flip-flopped.
Figure 2 · V2's architecture was flawless on paper, died from stuffing everything in
Death #3: Adding an orchestration layer — the pressure just moved house
What we did
Added an "orchestration" layer before the core agent: packed tool description corpus into a knowledge base, used a small model to pick capabilities based on the user query, then handed the narrowed set to the core agent for LLM reasoning.
user query → orchestration (small model picks capabilities) → core agent (LLM reasons)
Result
Capabilities were indeed decoupled. The core agent got a cleaner tool list, and the prompt no longer had to include all tools.
But the problem didn't disappear — it moved from core agent to orchestration layer:
- If orchestration picked wrong, everything downstream was wrong (and more insidiously — the core agent would work very hard with the wrong tool, producing output that looks professional)
- Maintaining the orchestration corpus became a new bottleneck
Lesson
Moving complexity forward isn't solving it; it's just relocating it.
After that version we realized: the real solution isn't "who picks the tools," it's how to make the candidate pool small enough and the selection deterministic enough.
Death #4 (the turning point): Layered governance, stability first
V4's core words: layered governance, stability first, large-scale noise reduction, hallucination control, cost control.
This version didn't change the architecture; it changed where the constraints live.
Specifics involve many internal designs — how tool descriptions are graded, how recall strategies are configured, how much orchestration should converge — this is my bread and butter, so I won't detail each one. But the thinking can be explained, and thinking is more valuable than parameters:
- The same tool information shouldn't be maintained in just one version. Retrieval needs "detail" to compute similarity accurately; selection needs "concise" so it doesn't blow up the context. Using one corpus to serve both is inherently contradictory — you have to split by purpose, and make both sides align.
- Changing configuration shouldn't require a release. Tool description text is an operational asset, not code. It must be hot-updatable, or every description tweak needs a release window, and "operability" is always a negative score.
- The candidate pool must be narrowed before model reasoning. Not because the model can't choose, but because the candidate pool itself is the breeding ground for hallucinations — the larger the pool, the more the model tends to "guess at something that looks right."
- Memory can't be used as-is. We stepped on this for a long time before we got it; more below.
- Deterministic things should never go to the model. Fully deterministic chains get hardcoded returns; semi-deterministic ones (like extracting an order number from a conversation) don't go through reasoning either, but go through routing. The model only handles genuinely uncertain parts.
Why memory can't be used as-is
A concrete example. An anomaly list — the tool returns a big JSON. Previously we saved the JSON verbatim into memory. Three things happened at once:
- The model saw a number in the JSON and treated it as an anomaly order number (this is the #1 source of "fabricated parameters")
- Tokens burned for nothing
- Coreference resolution became very hard — the model couldn't distinguish "this is item 3 in the list" from "this is the current order"
There's only one fix: when storing, don't store the raw structure. Concatenate the fields into a semantic summary and store that.
One move solves three problems: prevents hallucinations, saves tokens, aids coreference.
Results
- About 90% of hallucinations eliminated
- Response time dropped from over 15 seconds to under 10 seconds
Note: "about 90%," not "eliminated." I'll say what's left later.
Hallucination isn't one problem; it's six
Most people say "don't fabricate" when talking about hallucinations. Useless.
Because hallucination isn't one problem — it's six. The root causes are different, and so are the remedies.
| # | Type | Behavior |
|---|---|---|
| 1 | Skipping tool | Should call a tool, but answers directly from memory, returns immediately |
| 2 | Fabricating parameters | No anomaly order found, but there's a number-like type in context, so it assigns it as the order number (ticket ID, user ID) |
| 3 | False success | Tool call fails but reports success — the report API returns an error, but it returns "reported successfully" |
| 4 | Cross-turn contamination | Treats A order's diagnosis as B order's |
| 5 | Cross-object contamination | Same, pollution across business objects |
| 6 | Routing hallucination | Calls a tool that doesn't exist at all |
This table is worth printing and pasting on the wall. Because many teams are using "add forbidden words" as the one cure for six different diseases.
Figure 3 · The six hallucination entry points: they live in different places in the reasoning chain
There's an order to governance, and it can't be reversed
Architecture layer: strip deterministic chains, early exit, shrink candidate pool
Input constraints: schema rules, parameter format constraints, memory summaries, forced tool whitelist
Process control: orchestration layer pre-narrows the tool set
Bottom safety net: continuous evaluation (real-time sampling + offline replay)
The order matters: first make it impossible for hallucination at the architecture level, then tighten the opening with input constraints, then limit the choice space with process control, and only then back it up with evaluation.
Many teams do the reverse — they go straight for prompt forbidden words. Forbidden words can suppress "tone," but not "facts."
Figure 4 · The order of four governance layers: the deeper you go, the cheaper; the higher, the more expensive
Three actions that actually worked
One: make the model see examples of "using a tool to complete a task" in context.
Initially, our memory stored only results, not tool calls, because we thought it was redundant with the model's output. Result: the model learned to just reply in text — it never saw any precedent for "getting things done via tools," only "replying with words."
This is a classic pit: you train it to see, so it becomes what it sees. In scenarios where tools should be called, the memory must retain the traces of tool calls.
Two: write extraction rules into the parameter schema.
For example, anomaly order numbers: how many digits, where to find them in context, where to find them in historical memory. The model doesn't know your ID format — if you don't write it clearly, it will make one up to fit.
Three: make the candidate tools basically fixed before reasoning starts.
Which tools can be used in the ReAct stage should ideally be decided at the orchestration stage. If during reasoning a tool is wanted but not in the candidate pool, the correct response is to retry or ask, not to let it through.
Early exit: the biggest step in hallucination reduction isn't making the model smarter
ReAct's rhythm is: model → call tool → back to model → decide again.
But some tool calls return a terminal result; no further model reasoning is needed.
For example, pushing a processing-result card to the user — the returned result is already the final state; there's no need to feed the subagent result back into the model to think about "how to reply."
At that point, end directly: saves tokens, and one less chance for hallucination.
Combined with deterministic chain stripping, this is the main source of "90% hallucination reduction, 15s→10s."
It's not about making the model smarter; it's about not letting the model do things it shouldn't do. I consider this the guiding principle of Agent engineering.
Evaluation: how to prove the judge itself didn't misjudge
One principle
What rules can judge, don't give to the model; what you give to the model must have anchors, reference answers, evidence, and be verifiable.
Split into three layers per this principle:
| Layer | Scenario | Practice |
|---|---|---|
| L1 | Rules can judge | Hardcode the check, don't go through the model. E.g., called a tool that shouldn't be called, didn't report something that should be reported |
| L2 | Needs semantic understanding | Check against a reference. Must give the judge anchors and tool-return evidence, no free play |
| L3 | The judge itself is unsure | Abandon single-shot judgment, use repeated sampling to converge |
Evaluation will always overfit
The judge follows the standard answer, scores get higher and higher, and at launch it falls apart.
This is the sneakiest failure mode of an evaluation system — it doesn't error out; it just makes you more and more confident.
The only fix is structural: when building the evaluation set, tag by intent domain to reduce cross-domain interference; the data used for diagnosis and the data used for final acceptance must be two separate sets, physically non-overlapping.
If you use the same data for diagnosis and scoring, the model evolves toward that data, looks more and more accurate, and learns nothing.
Current state
In evaluation, humans still make up 80% . That's our most unglamorous number right now, and the next thing to tackle.
Self-evolution: plan written, but not yet implemented
We want to build a closed loop: log collection → automatic grading → layered diagnosis → prescribe fix → human review → generate new version → re-evaluate → canary → real-time traffic validation.
I won't detail the chain (it's not running, and writing it would be empty talk). But I can give one judgment now:
The hardest part of self-evolution isn't "finding problems," it's "changing safely."
A system that can automatically modify prompts, if it can't quickly roll back when it breaks something, is a liability, not an asset. So every link in this chain must be able to brake — that's the real reason it hasn't shipped yet.
Directly editing the prompt — aren't you afraid of breaking it?
Afraid. But fear doesn't mean don't edit; it means add constraints to it.
The premise is that the prompt must be concatenated in segments, not one giant blob. Role, boundary rules, tool descriptions, historical memory, real-time context, forced injections — each segment has a clear responsibility, so problems can be localized to a segment.
Under that premise, a few hard rules:
- Only a very small number of segments can be touched at once, with limited word count — forces you to change precisely, not excessively
- Every change is recorded with a diff, snapshots are kept, and rollback is always possible
- Must go through canary, must have one-click rollback
- If consecutive challenge failures hit a threshold, forced cooldown — locked for a period, no more changes, with alerts to the group
The last one is the most counterintuitive and the most effective: give the system a "cooling-off period." People in a hurry will keep making random changes, the more they change the worse it gets, and eventually no one knows which version broke it.
Key realization: don't treat the prompt as code; treat it as configuration + experiment.
What to do when a tool call fails: not an abnormal termination
All tool calls are wrapped uniformly, catching exceptions, throwing custom exceptions (with the tool-returned error code). Then handle two cases:
Retryable (timeout, network jitter, IO exceptions): automatic backoff retry a few times, if still failing return a fallback. These retries don't consume model resources, because retry doesn't need the model.
Non-retryable: feed it back to the model as a tool response, and let the model decide — change parameters? Switch tools? Or fallback reply.
This is the essence of ReAct: failure is information, not a terminal state. What you need to do is correctly express the failure to the model, rather than making the decision for it or crashing it.
Subagent exceptions are handled the same: catch globally but don't throw, convert to a structured failure signal, and the main agent decides whether to switch domains, ask, or fallback.
One more layer outward: final safety nets like a turn cap and a thinking-timeout cap; when hit, output fallback text, with monitoring alerting and degradation strategies. This layer isn't to improve effectiveness; it's to guarantee the system never hangs indefinitely.
Memory: not about storing as much as possible
A few conclusions, no implementation details:
- In-session memory must have a cap, with asynchronous compression. Context isn't a free resource; when it hits a threshold, you must compress.
- Across sessions, only summaries are needed, not details. We haven't done long-term memory yet — not because we can't, but because we haven't figured out how much it's worth in a customer-service scenario.
- The write format of memory matters more than how much you store. The earlier point about "not storing raw structure, concatenate into semantic summary" is the most concrete embodiment of this.
Problems that still exist (we don't pretend to solve them)
- RAG recall's false-positive and miss rates — if recall is wrong, even everything downstream correct is useless
- Hallucinations still exist, just much smaller than before
- Evaluation is 80% human
- Self-evolution isn't implemented yet — the plan is written, but it's not running
I write this not to seem honest. It's because an Agent team that says it has solved everything usually has only built a demo.
Next direction
We're trying a more "deterministic" route: not using a generative model for filtering decisions, but using deterministic filtering + scoring + multi-select, outputting only within a given candidate pool, scoring only without writing answers. Cost and latency are an order of magnitude lower than an LLM, and it's type-safe.
Used in two places: a post-recall filter for "does the query match the tool," and a multi-select within the narrowed candidate pool. The original orchestration model degrades to a fallback, handling only low-confidence and complex queries.
Results aren't validated yet; after validation, I'll write another post. I'd rather give my judgment first, then the conclusion — the reverse becomes selling a solution.
Finally: the real methodology of all this
Looking back at four versions, the real progress isn't any technical choice, but learning to surface uncertainty and grade it:
- Fully deterministic → hardcode, don't go through the model
- Semi-deterministic → use routing, not reasoning
- Uncertain → only then give to the model for reasoning, and require anchors, evidence, and verifiability
This thinking explains three seemingly unrelated modules: hallucination governance, evaluation judging, and self-evolution.
Figure 5 · Surface uncertainty and grade it
If I can leave one sentence to teams building Agent deployments: don't ask "how to make the model smarter," ask "what exactly requires the model to do this?"
About the details
Some will ask: how exactly to grade tool descriptions? What are the recall thresholds? How does judging vote? What's the word limit for prompt edits?
I really left those out. The reason is simple: these numbers are what I've validated in real business. Outside that scenario, copying them is probably wrong.
If you're working on something similar and stuck in any of hallucination, cost, latency, evaluation — send me your specific scenario, and I can give my judgment directly, which is more useful than a parameter table.
Done: built an end-user AI Agent "Business Assistant" from 0 to 1 (200K+ users / 500K+ daily conversations), a business operations low-code platform from 0 to 1 (new scenarios from week-level to day-level), and funding-safety with 5x peak and zero capital loss.
Top 1 of 2 from juejin.cn, machine-translated. The original thread is authoritative.
Hmm, feels like a sudden enlightenment.
Thanks for the support.