跪拜 Guibai
← All articles
Artificial Intelligence

Building a Business Benchmark for Agents: Meituan's Merchant Assistant Inspection Loop

By 美团技术团队 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Most agent evaluation today leans on public Benchmarks that measure 'answer quality' but miss the operational checks a production agent must pass: correct tool calls, faithful data use, rule adherence, and stable execution. Meituan's practice shows a workable path for building a business-specific evaluation loop that catches failures and drives concrete improvements, with the honest caveat that offline gains don't automatically translate to business value.

Summary

Meituan's engineering team documents the full process of building an evaluation system for an agent that performs daily inspection on merchant advertising accounts. The work starts from a bottom-up decomposition of the business SOP and a top-down collection of real Bad Cases, resulting in 41 evaluation scenarios across six categories. Each scenario becomes an executable Task with a defined success criterion, a Rubric that separates must-pass checks from quality diagnostics, and a mix of deterministic Graders and LLM Rubrics for judgment.

A central finding is that evaluators themselves need calibration. The team cites OpenAI's audit of SWE-bench Verified, where 59.4% of tasks had defects, to argue that even widely used Benchmarks are unreliable. In their own system, they upgraded tool-call sequence checks from linear order to a DAG representation to avoid false negatives on legitimate parallel calls. They then used layered metrics to narrow down root causes, showing how a task failure is only a starting point for diagnosis.

The practical optimizations that came out of the error analysis follow a clear principle: reduce the deterministic constraints a model must memorize. Platform and account red lines moved into common configuration, business branches and stop conditions were written into Skills, and parsing and calculation went to deterministic code. After these changes, the same Benchmark showed improvements in capability ceiling and efficiency, though the team notes Pass^k still lagged Pass@k, meaning the bottleneck shifted from capability to consistency.

Takeaways
— Meituan built 41 evaluation tasks for a KA daily inspection scenario, derived from a business SOP decomposition and real Bad Cases.
— The evaluation system uses a mix of deterministic Graders for precisely checkable items and LLM Rubrics for semantic judgments.
— A Rubric separates must-pass items (correct tool, parameter consistency, output completeness) from quality diagnostics (efficiency, conclusion clarity) that don't affect task success.
— OpenAI's audit found 59.4% of 138 SWE-bench Verified tasks had defects, motivating continuous evaluator calibration.
— Early linear-order tool call checks wrongly flagged legitimate parallel calls as failures; upgrading to a DAG eliminated those false negatives.
— After optimization, the agent's capability ceiling and efficiency improved, but Pass^k remained significantly lower than Pass@k, indicating a consistency bottleneck.
— The team emphasizes that evaluator accuracy is distinct from agent capability: a lower misjudgment rate doesn't mean the agent improved.
— Offline Benchmarks were treated as necessary but insufficient evidence; online process metrics and business outcomes were monitored separately and cautiously.
— The Turing platform provides Task set management, environment configuration, item-by-item judgment, and manual quality inspection to support repeatable evaluation.
— Meituan's final advice: derive metrics from business risks, ensure metrics can both measure and diagnose, and iterate evaluator and agent as separate loops.
Conclusions

The shift from linear tool-call order to DAG is a practical refinement many agent evaluators will likely need as workflows become more concurrent and branching.

Separating 'doesn't know how' from 'can't do it consistently' is a useful diagnostic lens; the team's Pass^k vs Pass@k gap shows that reliability, not raw capability, is often the harder problem.

Meituan's principle of moving deterministic constraints out of the model and into code is a direct counter to the temptation to let LLMs handle everything, and it suggests a clear division of labor that many agent teams may want to adopt.

The cautious treatment of online metrics, acknowledging seasonality and lack of a control group, is a refreshingly honest stance compared to typical engineering blog claims of success.

Concepts & terms
Pass@k
A metric measuring the probability that at least one of k sampling attempts succeeds, indicating the agent's capability ceiling under multiple tries.
Pass^k
A metric measuring the probability that all k attempts succeed, indicating execution stability or consistency across repeated runs.
Grader
A component that judges whether an agent's output or execution satisfies a defined criterion. Deterministic Graders use exact rules; LLM Graders use semantic evaluation.
Rubric
A structured set of check items defining what counts as success for a task, often separating required items from quality diagnostics.
Tool Calls DAG
A directed acyclic graph representation of expected tool call sequences, where nodes are necessary calls and edges represent true data dependencies, allowing parallel or alternative orders without false failures.
Trace
The record of an agent's execution, including tool calls, parameters, and data returned, used to verify what actually happened during a task.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗