When an Analysis Agent Drifts: Dewu's Trace-to-Regression Loop for Verifying Fixes
Agent output quality becomes measurable and fixable the way ordinary software is, instead of a matter of opinion. The trace-coupled feedback loop, fixed-evidence Mock replay, and a judge that returns only a number add up to a reusable pattern for proving whether a change made an LLM tool better or worse.
When the analysis Agent at Dewu's load-testing platform starts drifting, the fault could sit in evidence collection, tool invocation, rule validation, or model reasoning. The final answer alone won't reveal which. The team's governance loop starts with user feedback carrying a Langfuse trace ID; the trace restores the full execution chain, and the fix lands in skills, rules, or evidence logic. Each confirmed problem becomes a reusable regression sample.
Evaluation freezes the problem scene. CLI Mock replays the same evidence the original case saw, the real Agent re-runs its tool chain and analysis, and an LLM judge scores the output on a five-tier scale (0, 0.25, 0.5, 0.75, 1) against a human-proofread reference answer. The reference answer is a baseline, not the only valid answer, and the judge returns only a number, never a rationale. Comparing Agent versions fixes the dataset, mock evidence, model, and budget; comparing models fixes the Agent and swaps only the model. Averages alone are not enough, so P95 latency, cost variance, and repeated runs get reviewed.
The Dubbo case shows the payoff. Manual troubleshooting experience was encoded into component knowledge, validation rules, and an analysis flow: Python handles deterministic checks (JMX parsing, service contracts, candidate generation), and the LLM reads component knowledge and confirms or corrects the diagnosis with the evidence. The Dubbo special set rose from about 50% to 90% solve rate, and the overall set went from 61.67% to 96.15% while average latency fell from 175 seconds to 113 and average trace cost dropped from about $0.587 to $0.198.
The non-monotonic run history (round 006 scored worse while being faster and cheaper) is the strongest argument in the piece for measuring cost, latency, and quality together; a single metric would have sent the wrong signal.
Scores are reported as exact numbers with no rationale because the judge prompt forbids explanations. That keeps the evaluation clean but pushes explanation back to manual case review, so low-score debugging stays a human task.
Quality, latency, and cost data were collected, but cost for two of the three candidate models showed as 0.00 with billing unconfirmed. The team still published the comparison, which suggests cost accounting for LLM agents lags behind quality measurement in practice.
The evaluation section is very practical. Two practical additions: for the fixed evidence in regression testing, it's best to snapshot the prompt and tool contract as well; otherwise, when you rerun a week later, the baseline has drifted on its own, and if the score drops, you can't tell who changed it. Also, I suggest reserving a dedicated set of 'incomplete evidence' cases in the evaluation set, such as tool timeouts or pagination returning only the first page, to see whether the Agent honestly reports it didn't get the data or fabricates a plausible-sounding conclusion—the latter is the failure hardest to detect in production.