跪拜 Guibai
← All articles
Architecture · Testing · Programmer

When an Analysis Agent Drifts: Dewu's Trace-to-Regression Loop for Verifying Fixes

By 得物技术 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Agent output quality becomes measurable and fixable the way ordinary software is, instead of a matter of opinion. The trace-coupled feedback loop, fixed-evidence Mock replay, and a judge that returns only a number add up to a reusable pattern for proving whether a change made an LLM tool better or worse.

Summary

When the analysis Agent at Dewu's load-testing platform starts drifting, the fault could sit in evidence collection, tool invocation, rule validation, or model reasoning. The final answer alone won't reveal which. The team's governance loop starts with user feedback carrying a Langfuse trace ID; the trace restores the full execution chain, and the fix lands in skills, rules, or evidence logic. Each confirmed problem becomes a reusable regression sample.

Evaluation freezes the problem scene. CLI Mock replays the same evidence the original case saw, the real Agent re-runs its tool chain and analysis, and an LLM judge scores the output on a five-tier scale (0, 0.25, 0.5, 0.75, 1) against a human-proofread reference answer. The reference answer is a baseline, not the only valid answer, and the judge returns only a number, never a rationale. Comparing Agent versions fixes the dataset, mock evidence, model, and budget; comparing models fixes the Agent and swaps only the model. Averages alone are not enough, so P95 latency, cost variance, and repeated runs get reviewed.

The Dubbo case shows the payoff. Manual troubleshooting experience was encoded into component knowledge, validation rules, and an analysis flow: Python handles deterministic checks (JMX parsing, service contracts, candidate generation), and the LLM reads component knowledge and confirms or corrects the diagnosis with the evidence. The Dubbo special set rose from about 50% to 90% solve rate, and the overall set went from 61.67% to 96.15% while average latency fell from 175 seconds to 113 and average trace cost dropped from about $0.587 to $0.198.

Takeaways
— A drifted analysis can trace back to evidence collection, tool invocation, rule validation, or model reasoning; the final output alone doesn't localize the fault.
— Feedback carries a langfuseId (trace ID), letting the team reopen the exact execution, check tool call order and evidence, and register the case in a governance queue.
— Regression testing freezes problem input and tool evidence while re-running the real Agent; the analysis process and final answer are executed fresh each time.
— LLM-as-a-Judge scores on five tiers (0, 0.25, 0.5, 0.75, 1) against a human-proofread expected answer, with only a number returned, no explanation.
— On the deepseek-v4-flash 007 runs, solve rate climbed from 61.67% to 96.15%, average latency fell from 175 s to 113 s, and average trace cost dropped from $0.586780 to $0.198030.
— Round 006 ran faster and cheaper than 005 but scored lower (92.31% to 83.33%); round 007 recovered to 96.15%, so quality and cost get tracked together.
— Dubbo special-case solve rate went from roughly 50% to 90% once troubleshooting experience was encoded into component knowledge, validation rules, and analysis flow.
— CLI collects evidence, Python does deterministic checks (JMX parsing, service contracts, candidate generation), and the LLM confirms or corrects the diagnosis; a Python rule hit is not the final diagnosis.
— Model comparisons require the same dataset, Mock evidence, and evaluator on one Agent version, with repeated runs per model; qwen and glm cost showed as 0.00 with billing unconfirmed, so costs couldn't be compared.
Conclusions

The non-monotonic run history (round 006 scored worse while being faster and cheaper) is the strongest argument in the piece for measuring cost, latency, and quality together; a single metric would have sent the wrong signal.

Scores are reported as exact numbers with no rationale because the judge prompt forbids explanations. That keeps the evaluation clean but pushes explanation back to manual case review, so low-score debugging stays a human task.

Quality, latency, and cost data were collected, but cost for two of the three candidate models showed as 0.00 with billing unconfirmed. The team still published the comparison, which suggests cost accounting for LLM agents lags behind quality measurement in practice.

Concepts & terms
Langfuse
Open-source LLM observability and evaluation platform. Dewu uses its trace recording, dataset management, and scoring features to reconstruct agent executions and run offline evaluations.
LLM-as-a-Judge
Using a language model to score outputs. Here a strict prompt defines a five-tier scale, and the judge must return only a number, never explanations.
CLI Mock
A plugin that replays fixed tool evidence during regression testing, so different Agent versions see identical problem inputs and responses despite changing online data.
Annotation Queue
A Langfuse mechanism to collect and label user feedback records for later review, turning each reported problem into a source of regression cases.
From the discussion
Featured comments
星云PLUS团队

The evaluation section is very practical. Two practical additions: for the fixed evidence in regression testing, it's best to snapshot the prompt and tool contract as well; otherwise, when you rerun a week later, the baseline has drifted on its own, and if the score drops, you can't tell who changed it. Also, I suggest reserving a dedicated set of 'incomplete evidence' cases in the evaluation set, such as tool timeouts or pagination returning only the first page, to see whether the Agent honestly reports it didn't get the data or fabricates a plausible-sounding conclusion—the latter is the failure hardest to detect in production.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗