Agent Evaluation Is Moving From Grading Answers to Auditing Behavior
As Agent frameworks move from single-turn Q&A to multi-step task execution, evaluation that only checks final answers misranks unreliable, costly Agents as equal to stable ones. Teams building on Claude Code, OpenClaw, or internal Skill platforms need trace-level behavioral evaluation and binary-rubric alignment to turn subjective judgment into repeatable, automated quality gates.
Agent evaluation has outgrown leaderboard scores and demo impressions. Meituan's Turing team frames it as a precision instrument that must answer not just whether an Agent succeeded, but how — examining the planning path, tool-call stability, token cost, and safety risks in every run. The core formula is observation plus evaluation equals continuous iteration; without full-trace logging, most failures are invisible and therefore unfixable.
The team's central methodology is bridging, not piling up metrics. Business KPIs like DAU or retention cannot map directly to model capability scores, so they insert a task-system bridge layer. Subjective evaluation is aligned through a "dictator" model — one strong role forces product, ops, and engineering to converge on the same binary rubrics (yes/no/unknown), which Meituan used to push human-machine agreement from 62% to 92% on one project and to 99% on another.
Long-horizon Agents like Claude Code and OpenClaw are forcing the biggest shift. Evaluation is no longer about whether the reply sounds good; it is about whether the task got done and whether the execution trace is repeatable. The audience for evaluation tools is also expanding from a small circle of ML engineers to anyone who writes a Skill, which demands simpler, automated, pipeline-integrated evaluation infrastructure with full-link replay, sandboxed execution, regression gates, and AI grading engines that amplify a core evaluator's judgment rather than replacing it.
Meituan's framing of evaluation as a "precision measuring instrument" rather than a leaderboard exercise reflects a broader industrial reality: Agent reliability is an engineering problem, not a model-capability problem, and the bottleneck is observability.
The "dictator" model for aligning subjective evaluation is a deliberate rejection of democratic consensus in favor of speed and consistency — a trade-off that many Western teams resist but that Meituan credits for lifting human-machine agreement from 62% to 92%.
Binary-rubric decomposition (yes/no/unknown) treats the proportion of "unknown" verdicts as a leading indicator of rubric quality, turning ambiguity into a signal for where the evaluation framework itself needs redesign.
The observation that divergent expert opinions on what constitutes "good" can be productized as separate Agent strategy branches suggests a path to handling long-tail scenarios without forcing a single gold-standard answer.
Long-horizon Agents invert the evaluation cost structure: richer execution traces make human evaluation a bottleneck, which pushes the workflow from human-led to machine-led evaluation — but only after core evaluator standards are encoded into rubrics.
The Skill evaluation gap is not primarily technical; it is a tooling and literacy gap. As Skill creation drops to near-zero-code, the population that needs to evaluate output will include product managers and operators who have never run an eval before.