跪拜 Guibai
← Back to the summary

When an Analysis Agent Drifts: Dewu's Trace-to-Regression Loop for Verifying Fixes

Load-Test AI Agent Governance and Evaluation Tuning Practices

After the load-test intelligent analysis Agent was connected to real business, the analysis results drifting off-course could be caused by any one of: evidence collection, tool invocation, rule validation, or model reasoning. Looking only at the final conclusion makes it hard to locate the root cause, and also makes it impossible to verify whether a change is effective.

Day-to-day iteration after launch mainly needs to solve two problems:

Using Dubbo debugging as an example, this article walks through the complete process from user feedback, Trace troubleshooting, consolidating experience, to fixed-scenario regression testing, presenting how to continuously polish the Agent by combining Langfuse and CLI Mock.

I. Background: Bringing the Agent into the Debugging Workflow

Load-test debugging is the area where users report the most problems. The platform supports 10 common protocols and more than 30 script components. When newcomers encounter unfamiliar protocols, script orchestration, or parameter configuration, they often hit debugging errors. In the past, troubleshooting such problems required checking scenario configuration, JMX scripts, request/response, JMeter logs, and Trace one by one, a long and laborious process.

To address these debugging issues, we integrated the intelligent analysis Agent into the existing business workflow. It automatically collects scenario configuration, scripts, request/response, logs, Trace and other information, completes a preliminary diagnosis, and provides handling recommendations. When script modification is involved, the Agent first presents the specific changes, and writes them back to the platform only after user confirmation.

From the user's perspective, the analysis still originates from the debug page. Internally, the request passes through: Debug page Copilot → Java business gateway → Agent SDK runtime → Skill and CLI → platform data and local scripts. Rules and programs first organize evidence and generate candidate causes; the model then reviews with context and outputs a solution; the platform is responsible for approval, execution, and result display. Langfuse records the complete analysis process, providing the basis for later troubleshooting and evaluation.

II. Goal: From Launch Feedback to a Governance Closed Loop

After the Agent was rolled out to real business, the governance problems we encountered centered on three aspects:

Feedback is hard to collect: Users usually report problems after the analysis finishes, relying on post-hoc descriptions and manual communication. The input at the time, scenario configuration, and execution records are often missing, making it hard to fully restore the problem site. Evaluation is not standardized: Different scenarios, versions, and user concerns vary; without a unified evaluation standard, subjective feedback alone cannot objectively compare the Agent's actual effectiveness. Process is not transparent: Without visibility into the Agent's tool calls, evidence sources, and reasoning process, it is hard to confirm whether the problem lies in evidence collection, rule processing, or model analysis.

Addressing these problems, we wanted to link user feedback, Trace troubleshooting, problem fixing, and regression evaluation into a continuous governance process: user feedback → link Trace → restore and locate the problem → fix Skill, rules, or evidence collection logic → solidify regression cases → go live after evaluation passes.

Each confirmed problem should leave behind reusable troubleshooting experience and repeatable regression samples. This way, later versions can verify the effect of changes in the same problem context, avoiding judgment based on a single experience or subjective impression about whether the Agent truly improved.

III. Langfuse Governance Foundation: From Process Recording to Problem Troubleshooting

The previous section defined the governance closed loop from user feedback to regression evaluation. To make this workflow work, the Agent's analysis process must first be fully recorded, and user feedback, invocation evidence, and evaluation results linked to the same execution.

Langfuse does not directly participate in business diagnosis. It preserves the Agent's inputs/outputs, tool calls, execution duration, and model costs, so that a single analysis can be restored, inspected, and compared. When a user reports that the suggested solution did not resolve the problem, we can return to the analysis context at the time through the corresponding Trace and determine which segment the deviation occurred in.

Langfuse's Role in the Governance Closed Loop

The load-test platform is responsible for running the Agent and executing tools; Langfuse records the process and links governance data. Online Traces are used to troubleshoot specific problems; offline experiments are used to verify the quality of modified solutions. See Langfuse's official documentation for related capabilities.

From the system architecture, the SDK is responsible for reporting call data, Web/API receives events, Worker processes asynchronously, and the storage layer provides Trace query, dataset management, and scoring analysis capabilities. The relationships between components are shown in the figure below; the architecture references Langfuse's official general architecture. CLI Mock is the load-test platform's extension in the evaluation scenario and will be introduced in Chapter 4.

Feedback Case: From User Feedback to the Pending Governance Queue

Using a "the solution given by the Agent did not resolve the problem" feedback example, we show how to restore the analysis process through Trace. The feedback includes scenario information, the user's original request, solution shortcomings, and langfuseId (the Trace ID for this invocation).

① Locate the call: Find the corresponding record by Trace ID, and verify the call time, business scenario, user input, and Agent version to confirm the feedback matches the Trace. (left) ② Restore the chain: Examine the call order of the model and tools, and cross-check the user's request, the analysis process, and the final solution to confirm which steps the Agent actually executed. (right)

③ Verify evidence: Expand key tool nodes, check call parameters, execution results, and exception information, and determine whether the problem comes from missing evidence, tool failure, rule processing deviation, or the model failing to properly use existing evidence. (left/right) ④ Register the pending governance case: Record the solution shortcomings, preliminary cause, and content pending review; add it to the pending governance queue through Annotation Queue, as a source for later fixes and regression samples. The queue name follows the actual configuration; see the official documentation for operations.

Entering the pending governance queue only means the problem has been recorded. After cause confirmation and Agent fixes, it is still necessary to re-evaluate in the same problem context to verify whether the modification is effective. The next section explains how fixed evidence datasets and CLI Mock support this process.

IV. Evaluation System: Verifying Agent Quality with Fixed Evidence

The previous section covered restoring an analysis process through Trace. This section further explains how, after the Agent is modified, to verify whether the change is truly effective.

The basic principle of evaluation is to let different versions of the Agent face the same problem, business context, and diagnostic evidence, changing only the factors to be verified, then compare the quality, duration, and cost of the final solutions. Langfuse manages evaluation datasets, experiments, and scoring results; CLI Mock restores fixed business evidence during regression testing.

Fixing the Problem Context, Regression-Testing the Agent Solution

The platform solidifies user problems, scenario configuration, JMX scripts, logs, and request/response information into evaluation cases, and configures human-proofread reference answers for each case. During regression testing, CLI Mock returns evidence consistent with the original problem, avoiding online data changes affecting before/after comparisons of versions.

The evaluation still runs the real Agent. The Agent calls tools, reads Mock evidence, analyzes the problem, and generates a solution through the normal business flow, and the evaluator then scores the actual output. Therefore, what is fixed is the problem input and tool evidence; the Agent's analysis process and final answer are still re-executed.

When verifying Agent tuning effects, fix the dataset, Mock evidence, model, evaluator, and execution budget, modifying only the corresponding Skill, rules, or analysis flow. When comparing candidate models, fix the Agent implementation and evaluation configuration, switching only the model. For cases with high result variance, run repeatedly and observe multiple rounds, rather than judging based on a single highest score.

Evaluation flow: fix problem and evidence → Agent re-analyzes → automatic scoring → result comparison and regression.

Evaluation Metrics: Solution Quality, Latency, and Cost

Evaluation results observe the Agent's performance from three aspects: solution quality, response latency, and usage cost.

Quality is the primary criterion. When quality is close, latency and cost are then weighed. In formal comparisons, besides the average, attention should also be given to P95 latency, cost fluctuation, and repeated-run results, to avoid the average masking extreme cases.

Evaluator: Five-Tier Scoring Based on Human-Expected Answers

The evaluator uses model review (LLM-as-a-Judge): with the human-proofread expected answer as the primary basis, judging the semantic consistency and key content coverage of the Agent's output. It receives three types of input: input is the user problem, used to understand background, goals, and constraints; expectOutput is the human expected answer, providing the scoring baseline; actualOutput is the solution actually generated by the Agent, being the evaluation subject.

Scoring boundaries: the reference answer is not the only correct answer, and the actual output is not required to reproduce the reference answer's reasoning process, wording, structure, evidence order, or output style.

The evaluator returns only one legal score, without outputting reasoning or any other text. The human expected answer provides a unified standard and also defines the evaluation boundaries; low/high scores and disputed cases need to be sampled and reviewed, and after revising omitted valid solution paths, re-evaluate with a unified version. Online fix effects are confirmed by users through further debugging.

① Define the evaluator: Edit the required evaluator definition in Langfuse, defining clearly the variable information to pass.

You are a strict, objective answer-correctness evaluator.

Your task is to evaluate, based on the human-proofread reference answer, whether the Agent's actual output semantically completed the user's task. The reference answer is not the only correct answer, and the actual output is not required to reproduce the reference answer's reasoning process, wording, structure, evidence order, or output style.
Variable descriptions:
-<user_input>: the user's original problem, used to understand the task goal, context, constraints, and hard restrictions.
-<expected_output>: the human-proofread reference answer, the primary scoring basis.
-<actual_output>: the solution actually generated by the Agent, the subject of this evaluation.
Please complete the following analysis internally:
1. From <user_input>, identify the problem the user actually needs to solve, the goal, constraints, and hard restrictions.
2. From <expected_output>, extract scoring points and split them into two categories:
   - Primary scoring points: core conclusions, final solutions, necessary constraints, key facts, or key evidence required to solve the user's primary problem.
   - Secondary scoring points: auxiliary troubleshooting information, secondary evidence, supplementary explanations, additional examples, formatting instructions, expressions unique to the reference answer, or non-essential details.
3. Judge whether <actual_output> semantically covers these scoring points.
4. Judge whether <actual_output> solves the user's primary problem.
5. Judge which primary or secondary scoring points <actual_output> omits.
6. Judge whether <actual_output> contains content contradicting the reference answer's core conclusions, key facts, or the user's explicit constraints.
7. Choose the score according to the scoring criteria.
Scoring rules:
- Score primarily based on the primary scoring points in <expected_output>.
- If the actual output comprehensively solves the user's primary problem and covers primary scoring points, even if some secondary scoring points are missing, it should be considered a high score.
- When the actual output's expression, content order, reasoning path, or wording differs from the reference answer but is semantically consistent, no points should be deducted.
- The actual output is not required to reproduce the reference answer's reasoning chain, evidence order, or output style.
- Missing secondary scoring points, auxiliary information, optional explanations, or non-critical supplementary notes should not incur major deductions.
- If what is missing is only auxiliary troubleshooting information, secondary log explanations, or supplementary notes unique to the reference answer, and it does not affect solving the problem, it should be treated as a minor omission.
- If missing key evidence that directly affects the credibility, verifiability, or executability of the conclusion, the score must not exceed 0.75.
- Supplementary content in the actual output that is not in the reference answer does not raise the score; but if it does not contradict core conclusions, key facts, or the user's explicit constraints, and does not mislead the user, it is not deducted.
- Content contradicting core conclusions, key facts, or the user's explicit constraints should be deducted.
- If a requirement appears only in <expected_output>, not in <user_input>, and is not explicitly marked as a necessary constraint, that requirement can only be treated as reference information, not as a hard deduction item.
- If the user's original problem explicitly requires a certain output format, field, marker, action, or behavior, and the actual output does not satisfy it, deduct points.
- Do not use your own knowledge to replace or override the core conclusions of <expected_output>.
-<user_input> is only used to understand the task goal and constraints; it cannot serve as an independent standard answer.
Scoring criteria:
1:
The actual output comprehensively solves the user's primary problem, fully covers the primary scoring points in the reference answer, is semantically correct, and has no substantive errors, omissions, or contradictions. Secondary scoring points that do not affect solving the problem may be missing.
0.75:
The actual output has solved the user's primary problem, core conclusions are correct, and it covers most primary scoring points; only minor omissions, missing secondary evidence, incomplete expression, or small issues that do not affect the main conclusion exist.
0.5:
The actual output covers some primary scoring points and helps solve the user's primary problem, but has important content missing, incomplete answers, localized factual errors, localized contradictions, or fails to satisfy important constraints explicitly required by the user.
0.25:
The actual output covers only a few primary scoring points, with incomplete, unclear core conclusions, or obvious errors, providing only limited help in solving the user's primary problem.
0:
The actual output is contrary to the reference answer's core conclusions, irrelevant, empty, wrongly refuses to answer, or covers no valid expected content.
Scoring restrictions:
- If the actual output's core conclusion is contrary to the reference answer, the score must be 0.
- If the actual output does not solve the user's primary problem, the score must not exceed 0.25.
- If the actual output solves the user's primary problem but is missing important content, necessary steps, or important constraints explicitly required by the user, the score must not exceed 0.5.
- If the actual output comprehensively solves the user's primary problem and only misses secondary scoring points or has small issues not affecting the conclusion, the score should be 0.75 or 1.
- Only when the actual output comprehensively solves the user's primary problem, fully covers primary scoring points, and has no substantive errors or contradictions, can the score be 1.
- The score can only be chosen from 0, 0.25, 0.5, 0.75, 1; no other values may be returned.
The following is all the data to be evaluated:
<evaluation_data>
  <user_input>
  {{input}}
  </user_input>
  <expected_output>
  {{expectOutput}}
  </expected_output>
  <actual_output>
  {{actualOutput}}
  </actual_output>
</evaluation_data>
Only one of the following five values may be returned in the end:
0
0.25
0.5
0.75
1

Return of analysis processes, scoring reasons, text descriptions, tags, JSON, Markdown, or code blocks is prohibited.

② Add evaluation set data items: Find the data items that need evaluation in the Queue, edit the input and expected output needed for the evaluation request, and finally click to add to the evaluation set.

V. Tuning Case: Solidifying Dubbo Troubleshooting Experience, Improving Solution Quality

Measured Record: Quality Improvement Accompanied by Mid-Course Fluctuation

Let's first look at the staged results, then expand on the Agent-side specific improvements. The table shows the deepseek-v4-flash 007 series in the Langfuse run list; each round shows 16 items executed successfully. The scoring caliber and version information still need to be cross-checked against the original records. Execution success only means the analysis run completed.

From 001 to 007, the observed evaluation solve rate rose from 61.67% to 96.15%, average latency dropped from 175 seconds to 113 seconds, and average Trace cost dropped from $0.586780 to $0.198030. These are the differences between two rounds of records; attribution of tuning effects needs to be combined with the same samples, scoring rules, and version changes.

Figure: Solve rate, latency, and cost trends. The trend chart contains more historical records; the horizontal axis is not marked as a complete round sequence; the case values below follow the run list.

Figure 1 | Langfuse historical evaluation trends: latency, average cost, and Debug solve rate

The middle was not a continuous rise: 006 was faster and cheaper than 005, but the score fell from 92.31% to 83.33%; 007 rose back to 96.15%. Therefore we observe quality and cost simultaneously and return to low-scoring cases to analyze the cause. Using Dubbo as an example, we explain exactly where improvements landed in terms of knowledge and execution steps.

Dubbo Tuning Case: Solidifying Platform Troubleshooting Experience into the Agent

Dubbo debugging depends on environment, service, and method configuration, and also involves scenario orchestration and execution state. We distilled manual troubleshooting experience into the Agent's component knowledge, validation rules, and analysis flow, so that solutions land on specific configuration items, and verify effects through fixed-case regression testing.

First Organize: What Configuration Needs Checking in Dubbo Scenarios

Three Exception Examples: From Error Messages to Configuration Checks

① Consumer is shutting down → check addressing and exit reasons. The log marks dubbo://XXX:8848 as the provider; check whether Nacos was mistakenly treated as a direct connection address, and whether the registration protocol and namespace are correct; also combine JMeter logs to verify the task stop reason, not judging the root cause by "shutting down" alone.

② No provider available → check service discovery conditions. The example queries xxx/SpuXXXXService:1.0; check environment, namespace, group/version, and actual registration information item by item, then distinguish between configuration mismatch, instance unavailability, or route filtering. See the official troubleshooting documentation.

③ Failfast / $invoke → check method contracts. The request calls queryV6; the Provider's methods list includes V5 but not V6. Verify the target environment's methods, parameter types, and service version, and modify the configuration after confirmation; do not directly replace V6 with V5. See the generic invocation documentation.

Agent-Side Improvements: Knowledge, Rules, and Analysis Chain

CLI handles collection, Python handles deterministic processing, LLM handles reasoning. Python parses JMX, calls CLI to obtain contracts, executes validation and generates problem candidates, evidence indexes, and candidate reports; the LLM reads component knowledge and analysis guidance, combines evidence to confirm or correct candidates, and outputs root cause and modification solutions. Python rule hits do not equal the final diagnosis.

Regression Results: Dubbo Special-Purpose Improvement Boosts Overall Performance

Per the new version results provided this time, the Dubbo special-purpose evaluation rose from about 50% to 90%, an improvement of about 40 percentage points, an important source of overall improvement. The special-purpose, overall staged ranges, and the single-round 96.15% in section 5.1 correspond to different statistical scopes; when presenting, label the dataset and version separately.

Solution before optimization

Solution after optimization

VI. Model Comparison: Comparing Candidate Models Under Identical Conditions

Same-Stage Historical Result Comparison

The Agent's knowledge, rules, and execution flow all affect the analysis results, so the model itself also needs to be compared separately. This time, three candidate models were selected: deepseek-v4-flash, qwen3.8-flash, and glm5.3-flash. The table below organizes results from historical run records with experiment name "HTTP & Dubbo scenarios, round four", for observing same-stage performance. Since the dataset, Agent, and evaluator versions still need further verification, the following results are temporary stage references.

From this data set, DeepSeek has the highest solution score, and GLM has the lowest average latency. Qwen and GLM show a cost of 0.00, but currently it cannot be confirmed whether no cost was actually incurred or the billing data has not been integrated, so the three models' usage costs cannot be compared based on this.

These records only reflect the stage performance under the version at that time and are not yet sufficient to form formal model selection conclusions. For formal comparison, the same dataset, Mock evidence, and evaluator should be used on the same Agent version, with each model run repeatedly. Model selection looks at solution quality first; when quality is close, compare latency, cost, and multi-run stability.

The figure below preserves the original evaluation run list, for checking experiment names, execution times, and metric sources. The table in this section only counts the three candidate models above, not including other model records in the list.

Figure: Evaluation run records and model comparison data

Figure 2 | Evaluation run records: rounds, models, execution status, and metrics

VII. Experience Summary: Making Troubleshooting Experience a Verifiable Agent Capability

Use Trace to ground subjective feedback in concrete evidence. From user requests, tool inputs/outputs to final solutions, find which step the missed check, misreading, or execution failure occurred in, to determine the improvement location.

Hand codifiable experience to Python, semantic judgment to the LLM. Verifiable constraints such as environment addressing and service/method contracts are solidified into rules; the LLM combines business requests with raw evidence to review candidates and generate evidence-based solutions.

Keep fixed problems around to continuously verify later versions. Mock fixes the context, human expectations standardize the criteria, and evaluation simultaneously observes solution quality, latency, and cost. Then recover real effects through user debugging results, so each troubleshooting accumulates as reusable, verifiable capability for the next time.

Previous Articles

  1. Three Reversals of Multi-Agent Architecture: From Central Scheduling to Self-Driven Collaboration | Dewu Technology
  2. Jev Practice: Bringing a Lightweight Decision Layer to AI Workflows with 10x Speedup | Dewu Technology
  3. Domestic Computing Power · Ascend 910C LLM Inference Optimization (Part 2): Source Code Modification and Operator Optimization, 30% Speedup | Dewu Technology
  4. Domestic Computing Power · Ascend 910C LLM Inference Optimization (Part 1): Image Updates and Parameter Tuning, 40% Speedup | Dewu Technology
  5. Agent Practice with Dynamic Rules: Using Deterministic Engineering to Tame LLM Asset Intelligence Acceptance

Author: Hengyu

Follow Dewu Technology for technical content every Thursday.

If you find this article helpful, feel free to comment, share, and like.

Reproduction without Dewu Technology's permission is prohibited; violators will be held legally responsible.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

星云PLUS团队

The evaluation section is very practical. Two practical additions: for the fixed evidence in regression testing, it's best to snapshot the prompt and tool contract as well; otherwise, when you rerun a week later, the baseline has drifted on its own, and if the score drops, you can't tell who changed it. Also, I suggest reserving a dedicated set of 'incomplete evidence' cases in the evaluation set, such as tool timeouts or pagination returning only the first page, to see whether the Agent honestly reports it didn't get the data or fabricates a plausible-sounding conclusion—the latter is the failure hardest to detect in production.