跪拜 Guibai
← Back to the summary

Building a Business Benchmark for Agents: Meituan's Merchant Assistant Inspection Loop

The "Agent Evaluation Whitepaper" series focuses on how to put evaluation methods into practice in real projects. In the first article of the series, "Agent Evaluation Overview", we introduced the overall structure of the evaluation system and the two iterative loops where the evaluation system and the Agent promote each other.

As the business practice chapter of the series, this article uses the inspection task of a merchant AI assistant as a running case, introducing how to build a business Benchmark, calibrate evaluators and perform error analysis to simultaneously improve the evaluation system and enhance Agent capabilities. It also uses the Turing platform to support continuous regression and model selection, providing a reference for applying these methods to specific businesses.

1. Background

1.1 Business Background

The merchant advertising assistant is a business Agent designed for scenarios such as delivery inspection, anomaly diagnosis, data reporting, and plan editing. When executing tasks, it needs to read continuously changing real account data, call multiple tools, follow delivery rules, provide recommendations based on results, and adjust delivery plans directly when necessary.

Therefore, judging whether it can be used in production cannot rely only on whether the final answer "looks correct." The entire task must be verified: whether the intent and time range are understood accurately, whether data and tool calls are correct, whether key rules are followed, and whether the final account state or deliverable meets expectations.

1.2 Benchmarking the Industry

To build an evaluation system for the merchant advertising assistant, we surveyed representative Agent Benchmarks, focusing on three questions: how tasks are organized, how execution environments are constructed, and how results are judged.

The common approach of industry Benchmarks is to have the Agent complete full tasks in an executable environment, then check results through tests, validators, or Judges.

Industry Benchmark example

Public Benchmarks can help us design evaluations, but their scores cannot be directly used as the basis for launching the merchant advertising assistant, because the tasks, business rules, and data they cover differ from advertising delivery. We can reuse their task design, environment construction, and judgment methods, then build our own Task set around the advertising business, complete with execution environment, judgment mechanisms, and validation framework, to verify whether the Agent meets actual business requirements.

The following sections describe how to convert real business scenarios into a Task set, define success criteria and Rubrics, configure the execution environment, judgment mechanism, and evaluation protocol, and then drive Agent iteration through error analysis.

1.3 Construction Goals: Measurable, Trustworthy, Optimizable

We want this evaluation system to achieve three things:

  1. Measurable: Turn real business processes into executable, judgeable scenarios and success criteria.
  2. Trustworthy: Execute evaluations stably and at scale, with observable processes and reviewable conclusions.
  3. Optimizable: Locate root causes from evaluation results, verify changes, and gradually connect to online business performance.

2. Benchmark Design for the Merchant Advertising Assistant

The merchant advertising assistant needs to perform tasks such as inspection and diagnosis. When designing the Benchmark, first clarify which business scenarios to test and what counts as success in each scenario. Then translate these requirements into a Task set, execution environment, judgment mechanism, and evaluation protocol, so that different Agent versions can be compared under the same conditions.

2.1 Evaluation Scenario Design

When building an evaluation set, the first difficulty is often: which tasks should be used to test it?

Starting from online Bad Cases captures real problems, but samples tend to concentrate on errors that have already surfaced, making systematic coverage difficult; for Agents that are not yet online or have low usage, accumulated cases are also very limited. If the team proactively constructs evaluation samples, they must answer: which scenarios should be designed, and how to know whether key processes and risks have been covered, avoiding reliance only on personal experience?

After reflection and practice, we summarized two complementary paths: one that top-down decomposes from the business SOP to ensure core processes and key branches are not missed, and another that bottom-up adds coverage from Bad Cases in online usage and offline testing, covering boundaries and risks that are hard to foresee during design. Both types of scenarios together form a sustainable evaluation set.

Decomposing Evaluation Scenarios from the Business SOP

An evaluation scenario is essentially what questions we prepare to test the Agent with. Historical conversations only show how users asked in the past; core processes and key branches may not be fully covered. We first start from the business SOP, restoring each Skill into a business state diagram, identifying six types of elements: business states, data dependencies, conditional branches, early stopping points, capability boundaries, and handoff points across Skills. What results is not just a list of test cases, but a behavioral specification of "what should be done under what conditions, and when to stop or hand off."

Taking the daily inspection task of the local (KA) business as an example, its main flow is as follows:

Daily inspection main flow

This diagram should be understood as a "state-driven branch flow," not a fixed chain executed from start to finish:

The value of a business state diagram is turning "what should be done" from knowledge hidden in expert experience into a clear basis for designing evaluation scenarios.

Using Real Bad Cases to Fill Boundaries and Regressions

The business SOP describes expected behavior, but it cannot enumerate all unexpected failures of the Agent in actual execution. We collect Bad Cases from online usage and offline testing, first abstracting specific errors into reusable failure modes, then expanding them into boundary, anomaly, and regression scenarios.

For example, in a spot check of online tasks, we found that a merchant asked the Agent to inspect yesterday's delivery, and the Agent output a structurally complete report, but the Trace showed it actually queried today's data.

Date error case

In evaluation, this is not simply kept as a single wrong answer. It is abstracted into a "date understanding and calculation failure" and further covers today, yesterday, historical dates, weekends, and other cases.

Enriching Dimensions

Through top-down decomposition from the business SOP and bottom-up supplementation from real Bad Cases, we designed a total of 41 evaluation scenarios across six categories for the KA daily inspection. Core process scenarios ensure the main chain and key branches of daily inspection are covered; other groups supplement trigger word variants, business variables, and capability boundaries. Among these, routing and handoff are as important as the core main flow: a production-ready Agent must not only know "what should be done," but also when it should not start a full inspection, and instead ask, hand off, or refuse:

Scenario dimensions

2.2 Defining Success Criteria and Judgment Methods

Defining Success

Taking the KA inspection task as an example, when a merchant asks the Agent to "inspect yesterday's delivery," does receiving a report mean completion? We break this down from three aspects:

1. Does the execution flow follow the business SOP?

2. Are the execution process and conclusions evidence-based?

3. Is the final inspection report deliverable?

These check items constitute the scoring rubric for the task. Necessary steps, ordered calls, correct tool, parameter consistency, parameter compliance, output faithfulness, output completeness, and adherence to necessary output templates are required for success; indicators such as inefficient and duplicate tool calls, planning efficiency, and conclusion expression are mainly used for quality stratification and problem diagnosis, and do not affect task success.

When determining success, check items that can be precisely evidenced from the Trace are judged by deterministic Graders; parts that require understanding of business semantics and evidence relationships are left to the LLM Grader.

Evaluation Metrics

Passing one inspection does not mean the Agent is reliable: can it still complete tasks in another business scenario? If the same task runs multiple times, will it be inconsistent? How much time and cost does it take to complete? To answer these questions, we aggregate check results across scenarios and multiple executions, combined with runtime resource consumption, to form five dimensions of evaluation metrics:

Evaluation metrics

Among these, task completion rate reflects overall success; Pass@k and Pass^k measure capability ceiling and execution stability, respectively. Tool call order hit rate, necessary step coverage, and similar metrics help locate the main causes of success or failure. Planning efficiency, conclusion certainty, and other metrics observe output quality. Token usage, LLM call rounds, end-to-end latency, and TTFT measure resource consumption and production operational efficiency.

2.3 Evaluation Task Schema

Evaluation scenarios define "what to test," while success criteria define "what counts as doing it right." Organizing both into executable Tasks also requires clarifying prerequisites such as test accounts, run dates, business data states, and necessary permissions. For the 41 scenario instances in Section 2.1, we supplement each with Grader and execution environment prerequisites, finally forming 41 executable evaluation Tasks. The following shows a real Task from the KA daily inspection as an example to illustrate the complete Task structure:

Task schema example

In implementation, tool_calls_dag mainly corresponds to path planning, no_hallucination and hallucination.* correspond to tool and fact consistency, output_fields and output.* correspond to output quality, and execution.conditional is used to check conditional branches. Task completion rate, Pass@k, Pass^k, and cost/latency are aggregated by the Harness based on multiple Trials, and do not belong to the Grader of a single Task.

This Task uses a set of deterministic rules and semantic Rubrics to constrain the Agent's tool calls, parameter boundaries, factual basis, and final deliverable.

3. Evaluation Optimization and Agent Iteration (Error Analysis)

Error Analysis drives two iterative loops simultaneously: one corrects Tasks, Graders, and data to make evaluation more trustworthy; the other optimizes Skills, tools, and models to continuously improve the Agent. Case mining and attribution are the hub connecting the two loops.

Two iteration loops

3.1 Calibrating the Evaluator

The evaluator itself can make mistakes. When OpenAI audited 138 difficult SWE-bench Verified tasks, it found that 59.4% had defects in task descriptions or test design. This shows that even widely used Benchmarks require continuous calibration.

The same applies to the merchant Agent: if Trace parsing, deterministic Graders, or LLM Rubrics have biases, they can lead to incorrect attribution. Therefore, before optimizing the Agent based on evaluation, we must first ensure that the "ruler" is trustworthy.

Trustworthiness

Before large-scale use, evaluators must prove they can stably reproduce business experts' judgments. We build a calibration set confirmed by business experts, covering clear Good Cases, Bad Cases, and boundary Cases, and separately validate Trace parsing, deterministic Graders, and LLM Rubrics to avoid using an overall agreement rate to mask problems in different components.

The calibration focus for deterministic Graders is checking whether rules accurately express necessary business constraints. Early evaluation Tasks used a linear sequence to describe tool calls, which is suitable for simple precedences but easily judges reasonable parallel and merge paths as failures. After manual review found such false negatives, we upgraded complex paths to DAG: nodes represent necessary tool calls, edges only retain real data dependencies, and steps that do not depend on each other are allowed in different orders. After correcting the rules, we replayed historical samples to confirm that misjudgments were eliminated and no new issues were introduced.

DAG upgrade

The LLM Grader needs to be verified for consistency with business experts on semantic judgments. We manually spot-check the LLM Rubric, analyzing whether issues come from Rubric wording, missing evidence, or ambiguity in the business standard itself. For high-risk problems with insufficient evidence, the evaluator should also allow a "cannot judge" response rather than force a binary conclusion without basis.

It is important to distinguish: a reduction in evaluation misjudgment means the ruler has become more accurate, not that the Agent's capability has improved. Only by re-running the Benchmark with the same calibrated standards can we determine whether the Agent has truly improved.

Long-Term Trustworthiness

Initial calibration does not mean the evaluator will stay correct forever. Models, Agents, and Skills continuously iterate, business SOPs and capability boundaries change, and account states, tool interfaces, and Trace formats may drift. Therefore, evaluators need continuous maintenance just like the subjects being evaluated.

Continuous calibration does not mean permanently performing full manual review. Our practice combines three mechanisms:

Under this mechanism, human responsibilities are to calibrate standards, handle disputes, review high-risk results, and identify new failure modes. After confirmation, new problems are fed back into the calibration set, Grader, and historical regression samples, forming the evaluator's own continuous iteration loop.

3.2 Root Cause Analysis

Task failure is only a result, not the root cause. Task completion metrics tell us "whether it failed"; layered metrics further locate the problem to specific check items such as routing, necessary steps, tool parameters, conditional branches, factual basis, or final output.

Layered metrics

Atomic indicators can point to where the problem occurred, but root causes still need to be confirmed from execution evidence. The same "missing necessary step" could be because the Skill did not clearly constrain it, because the model missed a requirement in a long context, or because of data, environment, or Grader anomalies. After locating a specific check item, we directly inspect the local Trace, tool parameters, and return results associated with that metric, which usually confirms the cause and corresponding optimization point.

Only when metrics conflict with Trace conclusions, local evidence is insufficient, the same problem appears randomly, or the distinction between Skill and model issues remains unclear, do we further inspect the full Trace, repeat executions, or add control variable validation. This uses metrics to narrow the analysis scope while retaining Trace as the final basis for judgment, avoiding re-examining the full execution chain for every Bad Case and reducing the cost of problem localization.

3.3 Concrete Example: Inspection Task Optimization

After calibrating evaluators, we use the KA daily inspection as an example to show how to locate problems based on layered metrics, implement optimizations, and verify effects using the same Benchmark.

Locating the Problem

Layered metrics showed that daily inspection problems mainly concentrated in four areas: mixed configuration responsibilities, non-unique command routing, unclear conditional branches and stop conditions, and excessive reliance on the model for stable logic. After confirming root causes with related local Traces, we translated the problems into corresponding system optimization actions.

Problem localization and optimization actions

These optimizations follow the same principle: reduce the deterministic constraints that the model must memorize and freely judge. Platform, account, and parameter red lines are managed uniformly by common configuration; business branches and stop conditions are written into Skills; parsing, calculation, and filtering are delegated to deterministic code; the model focuses on process scheduling, condition understanding, and business expression.

Verifying the Optimization

After optimization, we used the same KA daily inspection Benchmark for regression. The main positive changes are as follows:

Optimization results

Note: (1) pp stands for percentage points (2) Optimization magnitudes have been desensitized.

The results show that the Agent's capability ceiling and execution efficiency both improved markedly, but Pass^k is still significantly lower than Pass@k, indicating the main bottleneck has shifted from "doesn't know how to do it" to "cannot do it consistently." This also guides our subsequent optimization thinking.

From Offline to Online

Offline Benchmarks can verify capability, stability, factuality, tokens, and latency, but they cannot directly equate to merchant business benefits. A complete evidence chain also requires validation through online process metrics and further association with user adoption, problem resolution, and delivery performance. We currently observe the following corresponding process metrics and business results:

Online metrics

Note: (1) Optimization magnitudes have been desensitized (2) Since the current number of merchants using it is small and there is no concurrent control group, these changes may also be affected by seasonality, promotions, and merchant operating states. We therefore treat them as preliminary signals of business value, not causal benefits; follow-up validation through larger samples and controlled experiments is still needed.

4. Engineering Implementation of the Evaluation System

The previous sections described the design of a business Benchmark and how to drive Agent iteration through error analysis. As models, Skills, and tools continuously update, teams need to reuse tasks and judgment standards, execute evaluations in batch, and retain evidence for problem review and version comparison. To this end, we use the Turing evaluation platform to organize evaluation preparation, execution judgment, and analysis regression into repeatable processes, reducing repetitive configuration and manual operations for each round of evaluation.

Turing evaluation platform

4.1 Building a Trustworthy Evaluation Process

Continuing with the "daily inspection" case, even if the Agent generates a complete report, the team still needs to confirm: did it query the correct date, perform necessary screening and drill-down, and draw conclusions based on real data? The success criteria and calibration methods defined earlier must be implemented in every round of evaluation. Turing supports "verifiable conditions, traceable conclusions, and correctable deviations" through task and environment management, item-by-item judgment, evidence drill-down, and manual quality inspection.

Trustworthy process

Unified Task and Configuration for Verifiable Baseline Comparisons

Comparing completion rates between two versions of the inspection first requires confirming they faced the same set of tasks and the same requirements. We use Task set management to maintain task inputs, prerequisites, and expected behavior, and use environment configuration to replicate the online Agent configuration as the evaluation baseline, clarifying models, Prompts, Skills, tools, and permissions. When validating a change, the team must clarify the changed items and verify that the rest of the configuration is consistent.

For example, to verify whether a new Skill version correctly handles "yesterday," reuse the same task and judgment standard, comparing the query date against the task context. Real account data changes over time, so the team also needs to verify run dates, account, and data states; replicating configuration does not fix the business data. Only when run conditions are comparable do changes in completion rate have explanatory power.

Turning Business Standards into Check Items with Traceable Judgment Evidence

In Turing's Evaluator, we configure Rubrics as executable check items: scripts judge items that can be precisely evidenced; LLMs judge items that involve semantic understanding. Items required for success determine whether a task is complete; quality diagnosis items are used for analysis and optimization, and are reused across multiple evaluation rounds with a unified standard.

During root cause analysis, the team can drill down from the evaluation report to a specific Task, failed check item, and execution snippet, verifying whether the date is correct and whether conclusions faithfully reflect data. Check items help locate deviations; root cause confirmation still requires upstream evidence, and when necessary, reproduction or comparative validation.

Integrating Manual Review into the Evaluation Process to Detect and Correct Misjudgments

We use Turing's manual quality inspection to spot-check LLM Judge results, using the human-machine agreement rate to measure consistency with business expert judgments, with emphasis on reviewing disputed samples and high-risk results. For inconsistent samples, we investigate Rubric wording, evidence understanding, or ambiguity in business standards, then verify improvements through historical sample regression, continuously discovering and correcting evaluation biases.

For example: a tool returns "delivery spend decreased," but the Agent writes in its report "delivery performance deteriorated." The LLM Judge marks it as faithful to the data, while the business expert considers that a decrease in spend is not sufficient evidence of deteriorated performance and marks it as not passed. For such disagreements, the team clarifies the requirement that "conclusions must be supported by data" in the Rubric, then uses historical samples to verify whether the human-machine agreement rate improves.

4.2 Evaluation Efficiency: From Single Evaluation to Continuous Regression

The merchant advertising assistant needs to be re-evaluated as models, Skills, and tools change. After the first evaluation, each subsequent iteration must answer: did the change solve the target problem, did other scenarios regress, and did cost and latency change? Turing retains Task sets, Evaluators, and Pipelines as reusable evaluation assets, allowing teams to follow existing processes to validate new versions, reducing repeated configuration, item-by-item execution, and result compilation.

Reusable evaluation assets

Reusing Existing Evaluation Assets to Reduce Prep Work

Building the inspection Benchmark for the first time requires configuring tasks, execution environments, check items, and calibrating evaluators. When changing models or adjusting Skills, if tasks and success criteria remain unchanged, the Task set, Evaluator, and Pipeline can be reused, focusing on updating the tested configuration and supplementing necessary scenarios.

Batch Sequential Execution and Judgment to Reduce Manual Operations and Data Handling

After preparation, Turing uses Pipelines to chain batch task execution, result collection, and machine evaluation, linking Tasks, execution environments, and Agent versions. Inspection reports, query parameters, tool returns, and Traces are collected during execution, and the configured Evaluator then performs item-by-item judgment, reducing the need to launch tasks one by one, copy execution results, and organize separate judgments.

Taking the adjustment of inspection stop conditions as an example, the team can reuse existing inspection tasks for batch validation, checking which tasks still show abnormal drill-down when no anomaly exists or continued checking after hitting a direct cause. Machines first apply unified rules for checks, then humans handle disputed results through random sampling and key reviews, allowing a new evaluation round to reuse the execution and judgment configuration from the previous round.

From Fixing Single Problems to Continuous Regression, Reducing Repeated Compilation and Validation

After fixing a failed task, we also need to check the impact of the change on other scenarios. Turing's evaluation report aggregates task completion, indicators across dimensions, latency, and token usage, and supports drill-down to specific Tasks, check items, and related Traces. Teams can narrow the investigation scope from scenario and check-item levels, then inspect corresponding evidence, avoiding repeated searching across scattered logs and result files.

For example, after making stop conditions explicit, we must verify both "do not continue drill-down when no anomaly" and "still complete necessary diagnosis when an anomaly exists." Under comparable conditions, the team reuses the Task set, Evaluator, and Pipeline to run regression, comparing target check items, overall completion rate, and cost/latency before and after optimization, confirming whether the change brings expected benefits and whether new regressions are introduced.

Thus, each iteration can continue validation on an existing foundation of tasks, standards, and evidence. Business personnel focus on new scenarios, abnormal results, and optimization judgments; the platform handles repeatable task organization, machine judgment, and result summarization, enabling evaluation to keep pace with the Agent's iteration rhythm.

5. Takeaway

Looking back at the entire practice, a business Agent evaluation system is not a test platform built once but a business infrastructure that starts from business tasks and gradually forms a closed loop of scenarios, metrics, evidence, diagnosis, and iteration. Other businesses do not need to copy our Tasks and metrics, but they can reuse this construction path.

6. Conclusion

When Agents move from "generating content" to "completing business tasks," evaluation must also shift from judging whether the answer is good to judging whether it completes the deliverable stably, along the correct path, based on real evidence, and at an acceptable cost.

A truly valuable business Agent evaluation system does not end with the number of Tasks, the number of metrics, or the level of automation. Its value lies in that business problems can be stably reproduced, failures can be quickly located, optimizations can be credibly verified, and online changes can continuously feed the next round of evaluation and business iteration.

Follow the "美团技术团队" WeChat account (meituantech) or visit: https://tech.meituan.com/ for more technical content.

QR code

This article is produced by Meituan Technical Team, with copyright belonging to Meituan. Non-commercial reproduction or use for sharing and exchange is welcome, please note "Content reproduced from Meituan Technical Team." Without permission, commercial reproduction or use is not allowed. For commercial use, please email [email protected] to request authorization.