Building a Business Benchmark for Agents: Meituan's Merchant Assistant Inspection Loop
Most agent evaluation today leans on public Benchmarks that measure 'answer quality' but miss the operational checks a production agent must pass: correct tool calls, faithful data use, rule adherence, and stable execution. Meituan's practice shows a workable path for building a business-specific evaluation loop that catches failures and drives concrete improvements, with the honest caveat that offline gains don't automatically translate to business value.
Meituan's engineering team documents the full process of building an evaluation system for an agent that performs daily inspection on merchant advertising accounts. The work starts from a bottom-up decomposition of the business SOP and a top-down collection of real Bad Cases, resulting in 41 evaluation scenarios across six categories. Each scenario becomes an executable Task with a defined success criterion, a Rubric that separates must-pass checks from quality diagnostics, and a mix of deterministic Graders and LLM Rubrics for judgment.
A central finding is that evaluators themselves need calibration. The team cites OpenAI's audit of SWE-bench Verified, where 59.4% of tasks had defects, to argue that even widely used Benchmarks are unreliable. In their own system, they upgraded tool-call sequence checks from linear order to a DAG representation to avoid false negatives on legitimate parallel calls. They then used layered metrics to narrow down root causes, showing how a task failure is only a starting point for diagnosis.
The practical optimizations that came out of the error analysis follow a clear principle: reduce the deterministic constraints a model must memorize. Platform and account red lines moved into common configuration, business branches and stop conditions were written into Skills, and parsing and calculation went to deterministic code. After these changes, the same Benchmark showed improvements in capability ceiling and efficiency, though the team notes Pass^k still lagged Pass@k, meaning the bottleneck shifted from capability to consistency.
The shift from linear tool-call order to DAG is a practical refinement many agent evaluators will likely need as workflows become more concurrent and branching.
Separating 'doesn't know how' from 'can't do it consistently' is a useful diagnostic lens; the team's Pass^k vs Pass@k gap shows that reliability, not raw capability, is often the harder problem.
Meituan's principle of moving deterministic constraints out of the model and into code is a direct counter to the temptation to let LLMs handle everything, and it suggests a clear division of labor that many agent teams may want to adopt.
The cautious treatment of online metrics, acknowledging seasonality and lack of a control group, is a refreshingly honest stance compared to typical engineering blog claims of success.