跪拜 Guibai
← Back to the summary

Agent Evaluation Is Moving From Grading Answers to Auditing Behavior

A Casual Discussion on Agent Evaluation — From Shallow to Deep, Explaining Agent Evaluation

Author: Meituan Technical Team Tags: Artificial Intelligence

This blog post is a popular science article that introduces Agent evaluation from the basics to the advanced. The first two chapters systematically introduce what evaluation is and how to establish an evaluation system. The second chapter is the practical experience summarized by the Meituan Turing Agent Evaluation Team through in-depth Business Partnering (BP) with various Meituan business teams — it is the knowledge we have gradually refined over two years of practice. The third chapter focuses on the changes that the emergence of long-horizon Agent frameworks like Lobster/Hermes has brought to evaluation.

After this blog post was published internally at Meituan, it received considerable attention. We found that everyone is very enthusiastic about Agent evaluation, so we decided to make the internal blog public, wanting to share these experiences with more colleagues, hoping to inspire or help everyone.

1. What is Agent Evaluation

1.1 The Core Purpose of Evaluation

The core purpose of evaluation is to answer: Is the Agent good? And specifically, where is it good, and where is it bad? Thus pointing the direction for the next iteration. Evaluation is the "precision measuring tool" for Agent effectiveness.

This is also why Agent evaluation cannot just stop at offline leaderboard submissions, nor can it just look at the performance of a single demo. It must serve the R&D, launch, regression, optimization, and large-scale implementation in real business.

The cornerstone of Agent evaluation is observation, hence we derive the Agent R&D formula — Observation + Evaluation = Continuous Iteration

1.2 The Difference Between Agent Evaluation and Traditional Model Evaluation

Evaluation methods change with the evolution of AI forms, roughly going through three stages:

Traditional machine learning was more about answering "is the calculation accurate?". Large model evaluation began to answer "is the model capability strong?". What Agent evaluation truly needs to answer is:

When a model is placed into a real system, coupled with Prompts, Skills, toolchains, memory, state management, and business processes, can it stably deliver good results?

This means the evaluation object for Agents is no longer a single model, but a complex system of "model + system + tools + process".

1.3 Why Agent Evaluation Looks Not Only at Results, But Also at Behavior and Process

In real scenarios, two Agents might both ultimately "do the right thing", but their engineering value is completely different:

If you only look at the final answer, these two would be misjudged as being at the same level; but from the perspectives of scaling, cost optimization, and user experience, the difference is very large.

Therefore, Agent evaluation needs to cover at least four layers:

In this sense, since the explosion of GPT in 2023, Agent development has rapidly evolved from ChatBot forms to multi-functional long-horizon Agents like Claude Code and OpenClaw. Agent capabilities are becoming increasingly powerful, and Agent evaluation is moving from "answer evaluation" to "behavior evaluation".

1.4 Why Observation is the Cornerstone of Evaluation

There is a simple understanding: Agents belong to the broad SaaS layer. Large models give Agents generalization capabilities but also introduce the problem of randomness, while users still expect a stable and reliable intelligent agent. To bridge the gap between "randomness" and "reliability", we must return to an engineering perspective on Agents — Problems that cannot be seen are almost impossible to solve stably.

A single execution of an Agent typically includes the following chain:

If any single layer within it has a problem, the final effect may degrade. For stable results, we need a stable process. But if the logging system can only see "what the user said" and "what was finally replied", it is almost impossible to determine the root cause of the problem. Precisely based on the simple problem of "I want to look at a Case but found no logs were printed", the industry developed Trace systems, which disclose the entire path of the internal logical reasoning process and record all input information that affects the model's output.

Precise observation of every "invisible action" of the Agent is the necessary path to leap from "probabilistic generation" to "industrial-grade reliability".

1.5 Summary: Trajectory Evaluation and Response Evaluation

Through the previous discussion, we know that Agent evaluation essentially answers whether it is good or not, pointing the direction for iteration. Agent evaluation itself needs to focus on both the result (Response) and the process (Trace, or Trajectory).

2. Core Methodology of Evaluation

2.1 The Core of an Evaluation System is Not Piling Up Metrics, But Building Bridges

Some colleagues might wonder, why must we "build bridges"? This is because an Agent evaluation system must pursue interpretability between business value and evaluation metrics. One of the difficulties of Agent evaluation is the natural gap between model capability metrics and business outcome metrics.

These two types of metrics cannot be directly mapped; there must be a layer of bridge metrics oriented towards the task system in between. We provide a layered approach as follows:

Taking AI search as an example, the business might care about DAU, retention, and clicks; the search system itself cares about recall rate and click-through rate; the Agent layer cares about whether intent recognition is accurate, whether retrieval is effective, and whether result integration is trustworthy.

Only by stringing these layers together can we truly answer "why did business metrics worsen" and "why did model capability improvement not bring business benefits". This must rely on people who truly understand the business process to jointly establish the metric system.

2.2 Objective Evaluation and Subjective Evaluation in Parallel

Through the introduction in Chapter 1, we know that the core goal of an Agent is to stably deliver good results. Agent evaluation in the industry has heavily borrowed from the methodology of large model evaluation, and can generally be broken down into objective evaluation and subjective evaluation:

Therefore, a more realistic approach is usually:

2.3 Using the "Human-Human Alignment, Human-Machine Alignment" Methodology to Align Subjective Evaluation

"Good or not" is a subjective question, and subjective standards need to be aligned, otherwise we cannot be sure whether an improvement after a certain iteration is a real effect improvement or just metric fluctuation.

In evaluation practice, what is truly difficult is not "no one can evaluate", but "different people evaluate differently, and machines and people evaluate differently". Over the past year, during our Turing team's deep BP process with business teams, we found that multiple teams successively fell into the same pitfalls.

The key knowledge accumulated by Turing Evaluation can be summarized as "Human-Human Alignment and Human-Machine Alignment", specifically as follows:

How to specifically perform alignment? The best practice is to drill down vague metrics into finer evaluation Rubrics (or evaluation dimensions: the academic community has not yet unified the terms evaluation dimension and evaluation rubric; we align here with the open-source project Arize AI), and then make each Rubric as binary as possible. Specifically:

The value of this decomposition method lies in: moving from "subjective vague feelings" to "judgeable factual evidence", reducing ambiguity through drill-down, thereby reducing disagreements between people and between people and machines. Applying this method, the human-machine consistency rate for the Digital Station Master reached 99%. Beam transformed its evaluation system using Turing's binarization scheme, and the human-machine consistency rate increased from 62% to 92%.

Below we share two cases.

Case 1: How to evaluate the quality of a middle school student's essay? (Full score 40)

Case 2: Taking whether the model reply in a rider outbound call scenario is "colloquial" as an example

Classic wrong demonstration — Please judge whether the large model's answer is colloquial, and score it from 0 to 10.

Improved version after drill-down and binarization:

Supplement: What is the relationship between annotation and evaluation?

Among them, annotation is an action, while evaluation is a set of goal-oriented judgment processes. Machine pre-annotation can help improve human efficiency, but it can only be called automated evaluation when the human-machine consistency rate is guaranteed; otherwise, it is just machine annotation.

2.4 Agent Evaluation is a Practical Science

From the execution chain perspective, Agent evaluation can be broken down into five key links:

These 5 links, together with online AB testing and continuous observation, constitute the data flywheel for Agent iteration.

Most teams new to Agent evaluation have a common misconception — they research from multiple sources to design a complex and exquisite evaluation metric system, but the more complex the metrics, the harder they are to execute and align.

Agent evaluation is a practical science. In the initial stage, "getting the data flywheel running efficiently" is far more meaningful than "designing a complex and exquisite evaluation system". The establishment of an evaluation system is not achieved overnight, but is fed by Good Cases and Bad Cases.

Therefore, the best practice path for building an Agent evaluation metric system is as follows:

Among these, the value of Bad Cases is often higher because they most easily expose capability boundaries and system weaknesses; the role of Good Cases is to help the team define the paradigm for high-quality completion.

The core capability of a mature evaluation team is not to build a perfect system from the start, but to continuously transform online problems, failure samples, and vague feedback into structured evaluation assets. We correct the goals of the evaluation system through Bad Cases and Good Cases, gradually correcting the negative deviation between the "dictator" and the true goal. For example, in the Fulfillment Digital Station Master business, there were only over 20 evaluation metrics at the start of the project, but after being fully rolled out over a year, we expanded to nearly 200 metrics.

2.5 Expert Knowledge Supplements Model Capability Deficiencies in Vertical Domain Scenarios

Since the release of GPT 3.5, although base model capabilities have undergone earth-shattering changes, model capabilities are ultimately not omnipotent. In the past three years of Agent practice, we have encountered a large number of unrealistic "wishful thinking" demands. We must know that models are trained; the model itself fits the probability distribution of Tokens. Even if capabilities "emerge" under large-scale parameters, the improvement of model capabilities still heavily depends on the input of corpus, especially high-quality corpus. Whether it's early RLHF or current algorithms like DPO and GRPO, although the training architecture is continuously simplified, the dependence on high-quality core data has never changed.

For example, ByteDance specifically established the crowdsourced expert annotation platform Xpert to produce high-quality data in professional fields such as geography, code, law, and medicine, thereby supporting the iterative training of the Doubao base model. The intuitive feeling of the improvement in base model capabilities is that it performs better and better in one vertical field after another.

Therefore, when we face the challenge of scarce business knowledge corpus (or no public high-quality data available) in a specific vertical domain, introducing expert knowledge input to supplement the model/Agent's capabilities becomes the key to breaking through — especially during the cold start phase of a project.

Echoing the previous text, the purpose of evaluation is to answer "Is the Agent good?", so who defines "good"? It relies on the industry experts who understand the business best and have the most sense.

2.6 Summary & FAQ

This chapter describes the core methodology precipitated by Turing Evaluation, starting from the Fulfillment project and further developed over more than a year of BP with various company business units.

FAQ

Q1: Must the "dictator" be one person, or is it enough for one team to follow a set of evaluation standards together?

First, a team needs to follow the same evaluation system. The role of the dictator is to solicit opinions from multiple parties and integrate the evaluation system. When project stakeholders cannot align their concepts, the dictator makes the final decision, avoiding the fragmentation of the evaluation system leading to stakeholders acting independently and the loss caused by internal friction.

Q2: Is there any risk in the evaluation system relying on a "dictator"?

We must recognize a fact: the evaluation system is constantly evolving. From business cold start to scale-up to full rollout, during this process, users shift from AI enthusiasts willing to try new things to all users, and the user profile will shift significantly. This causes the evaluation goals to need continuous adjustment as the business scales up. The dictator's value is more reflected in aligning standards; the evaluation goals are more driven and corrected by Bad Cases/Good Cases from real business scenarios. Of course, we need to select the person who understands the business best to formulate the evaluation system at the beginning of establishing evaluation standards.

Q3: Is it necessary to set up a seed evaluation set during the cold start phase? Can we just go live with a small traffic grayscale test and directly collect Good Cases and Bad Cases to drive evaluation?

Whether a seed evaluation set must be established during the cold start phase is essentially a trade-off between risk and cost.

Why it is recommended to set up a seed set: Large models have randomness, and Corner Cases may cause drastic shifts in Agent experience. Therefore, it is necessary to ensure the Agent's baseline capability through backtesting, and continuously incorporate Good Cases and Bad Cases to broaden the Agent's capability boundaries.

Regarding the strategy of small traffic grayscale launch: If the business scenario has a high fault tolerance rate, or the cost of building a high-quality seed set far exceeds the negative feedback from online trial and error, you can try going live with small traffic to collect online cases.

Recommended implementation plan: After manually producing a small number of evaluation sets, use AI to assist in generation or expansion, completing the construction of the seed evaluation set for cold start at a lower cost.

Q4: What should we do if the industry experts we invite have inconsistent definitions of "good"?

Answer: The most straightforward and effective method is to invite a group of industry experts to define the standard of "good", extract the common parts from them, and build the evaluation system. For example, invite top salespeople to define an excellent sales SOP.

So, are the non-common parts worthless? Not at all. The parts where expert opinions differ often mean that the business itself has multiple excellent strategies. We can transform these differences into different styles or strategy branches of the Agent (e.g., in AI telesales: the seasoned aggressive faction vs. the steady long-term faction), and allow them to be evaluated independently in different test sets (Benchmarks). These points of divergence are not noise, but rather important nutrients for the Agent's future refined iteration and coverage of more long-tail scenarios.

3. The Evolution of Agent Observation and Evaluation

GPT exploded in popularity in 2023, workflows emerged in 2024, Claude Code was released last year, and then this year's Lobster craze and Hermes craze brought long-horizon Agents gradually into the public eye. Agent Harness has also fully entered the era of long-horizon Agents.

The difference between a Long-horizon Agent and a Short-horizon Agent lies in how it handles the "complexity brought by time span":

What changes will these differences bring to observation and evaluation?

3.1 Observation and Evaluation in the Short-Horizon Agent Era

The evaluation objects in the short-horizon Agent era were relatively simple. The typical input-output form in the ChatAgent era was: Query -> Answer.

The common characteristic of such scenarios is: the Agent is more about "answering questions" and performing a small number of system operations, rather than "entering the operating system to execute tasks". Typical application scenarios include AI search and customer service robots. The evaluation focus for this type of Agent usually falls on the answer itself, for example:

Over the past year of development, Turing has formed mature solutions, including human evaluation and machine evaluation. Some cases are as follows:

3.2 Observation and Evaluation of Long-Horizon Agents

3.2.1 Long-Horizon Agents Bring a Paradigm Shift in Evaluation

Long-horizon Agents solve not "answering a question", but "completing a complex task". It typically requires:

Let's review the goals of observation and evaluation, and look at long-horizon Agents with these goals in mind:

3.2.2 Skill Evaluation

Before discussing Skill evaluation, let's share our research on this round of Lobster/Skill craze after the 2026 Spring Festival.

Who is proposing the evaluation needs for Lobster and Skill (User profile of the Lobster/Skill craze from February to now)?

In summary, the current needs related to Lobster and Skill are mainly in broadly defined operational efficiency scenarios, which can be roughly divided into three categories:

The user scale is expanding from a small number of professional roles to a wider population:

This leads to a direct conclusion:

In the future, the people who need to evaluate will not just be a small group of product, operations, and R&D colleagues, but potentially everyone who creates, modifies, or integrates a Skill.

This raises new requirements for the evaluation system:

The Current Essential Pain Point

The essence lies in — people don't know how to write a good Skill, and there is a lack of tools to evaluate Skills throughout their entire lifecycle.

To facilitate understanding, we break down the full lifecycle of a Skill as follows:

In summary, the pain points of Skill evaluation can generally be broken down into three aspects:

Task-Oriented Evaluation

On January 9, 2026, Anthropic published a blog post demystifying AI Agent Evaluation, which for the first time mentioned Task-oriented long-horizon Agent evaluation. In the article, a Task is defined as "a single test with clear inputs and success criteria".

We have synthesized the definitions of Task from Anthropic and open-source software, simplified as follows:

The prompt defines our question/request, and the expected behavior defines the behavior we anticipate the Agent to achieve. When we send a prompt to the Agent in a formal or test environment, and obtain the real execution path of the long-horizon Agent through traces, we can get the (prompt - expected_behavior - trace) triplet, similar to the (query - ground_truth - answer) of short-horizon Agents, and evaluation can proceed.

3.2.3 Differences Between Long-Horizon Agent Evaluation and Short-Horizon Agent Evaluation

The differences between the two can be summarized as follows:

The most fundamental change is:

ChatAgent evaluation cares about "whether it speaks well", while long-horizon Agent evaluation cares about "whether the task was accomplished, and how it was accomplished".

3.2.4 Moving from Human-Led Evaluation to Machine-Led Evaluation

The common process in the ChatAgent era was: Core Evaluator Alignment -> Outsourcing Alignment -> Machine Evaluation Alignment

In long-horizon Agent scenarios, this chain has the opportunity to be significantly shortened, even skipping outsourcing alignment, directly entering: Core Evaluator Alignment -> Machine Evaluation Alignment -> Large-Scale Expansion

There are three main reasons:

This does not mean that humans are unimportant, but rather means:

In other words, what AI evaluation truly aims to amplify is not "machine scoring" itself, but the judgment standards of the core evaluators.

3.2.5 What Capabilities Should Long-Horizon Agent Evaluation Infrastructure At Least Possess

If it is to support a large-scale Agent and Skill ecosystem within a company in the future, the evaluation infrastructure should at least include the following capabilities:

If these capabilities are lacking, evaluation easily remains at the level of "single analysis" and "project-based support", unable to truly become part of the production system.

4. Summary: Agent Evaluation is Moving from a Scoring Action to Infrastructure Capability

Looking at the big picture, with the enhancement of large model capabilities and the continuous evolution of Agent Harness, the evolution of Agent evaluation can be summarized in two sentences:

First, the evaluation object has changed

In the past, we evaluated "answers"; now we evaluate "task systems".

Therefore, we are no longer just concerned with the output content itself, but with the capability, stability, efficiency, and risk in the complete execution chain.

Second, the evaluation method has changed

In the past, the mainstream was text quality evaluation of Query -> Answer; now it is gradually shifting to behavior evaluation of Prompt -> Expected Behavior.

The standard answer is no longer always unique; process quality, task completion, and trajectory quality have become the new core objects.

Therefore, what is truly important in the future is not just whether we can do a few evaluations, but whether we can build an Agent evaluation system that can: see problems, articulate standards clearly, run at scale, integrate into processes, and drive iteration.

Appendix: Industry Research on Long-Horizon Agent Evaluation in the Open-Source Domain

In February 2026, when Lobster exploded globally, open-source software for Lobster evaluation emerged in the community:

Software Name Introduction GitStar Task Definition
pinchbench Open-sourced in February 2026, it is a performance benchmark system specifically designed for evaluating OpenClaw. Unlike traditional synthetic tests, PinchBench emphasizes "task simulation in real scenarios". It measures the real performance of models in handling multi-step work, actual development, and office environments by assigning AI agents complex tasks encountered in real work. 1200+ md file
claw eval Lobster Capability Evaluation released by Peking University task list 500 yaml file
WildClawBench WildClawBench: AI Agent Capability Evaluation in the Wild — Which of the 10 Major Models Has the Strongest "Lobster"? The core concept is "testing Agents in the wild" — not giving the model a carefully designed sandbox, but throwing it into the scenarios real users use OpenClaw Agent in every day, to see if it can survive. 500 Skill

| Follow the 'Meituan Technical Team' WeChat official account (meituantech) or visit: https://tech.meituan.com/, to read more technical干货!

| This article is produced by the Meituan Technical Team, and the copyright belongs to Meituan. You are welcome to reprint or use the content of this article for non-commercial purposes such as sharing and communication. Please indicate "Content reprinted from the Meituan Technical Team". This article may not be commercially reproduced or used without permission. For any commercial use, please send an email to [email protected] to apply for authorization.