Agent Benchmarks Don't Matter — Completion Rate, Speed, and Cost Do
Recently, models both domestic and international have been updating very quickly.
If you only look at launch events and leaderboards, everyone would think every model is strong. Bigger parameters, longer context, stronger reasoning, lower prices — it all sounds impressive.
But when you actually use them in a workflow, you discover something else: a model's strength isn't just about whether it can answer questions; it's also about whether it can complete a task from start to finish.
Especially in Agent scenarios.
A compound task requires a large model to call multiple tools. For example, asking a model to make a PPT isn't just about writing a few pages of text. In between, it needs to first understand the requirements, then search for information, read web pages, extract key information, organize it into a report structure, and if necessary, generate code or call plugins, finally producing a usable PPT file.
Below, I test two Agent tasks using the same prompt and the same Agent tool — Trae Work.
PPT Creation
Prompt:
Research the differentiated advantages and development paths of current mainstream short-video platforms, and organize it into a presentation deck. The research scope includes platform basics, user scale, content ecosystem, recommendation mechanisms, monetization models, and representative cases. Focus on comparing differences in user demographics, content types, and growth strategies across platforms, and summarize their successful experiences and future trends to provide a reference for product or market strategy.
Step 3.7 Flash
After receiving the instruction, Step 3.7 Flash analyzes the prompt's requirements, then searches corresponding website information, synthesizes the information, and finally calls the PPT plugin tool to create the slide file.
The style leans toward minimalism. It took about 5 minutes and cost roughly 1 yuan.
You can also clearly see what skills were used and what websites were searched for this task.
Overall, Step 3.7 Flash seems more like a production-grade choice.
Its advantage isn't necessarily that a single PPT page is the prettiest, but rather a balance between speed, tool invocation, and task completion rate. It's more suitable for high-frequency, multi-round Agent tasks that require stable delivery.
DeepSeek v4-Pro
DeepSeek v4-Pro followed the same path: identify the task, then find the tools needed — the PPT generation tool.
The color scheme was just a bit more vibrant. The final step also successfully called the PPT tool.
It took about 5 minutes, with token consumption around 0.5.
Simply put, DeepSeek V4's advantage is better content organization and presentation, suitable for scenarios that demand high-quality output. But for continuously running high-frequency Agent pipelines, you still need to look at end-to-end speed and per-task cost.
Minimax
Calling Minimax for this Agent task was a bit different. With the same prompt, Minimax's final step called an HTML tool to create the presentation. Normally, it should have called the PPT tool. Since it called the HTML generation tool, let's see how the result turned out.
The overall visual style of the HTML was decent. Because it's HTML, the code is easier to generate; if it were making a PPT, it might not be as well-controlled.
The style leans toward a fresh, clean look, and the data was fairly comprehensive. It took about 7 minutes, costing 0.7 yuan.
So in this test, MiniMax M3 showed good information organization and visual expression capabilities, but the controllability of tool selection still needs attention.
It's suitable for content pages, web reports, and lightweight presentation tasks; for strict office formats like PPT, Word, or Excel, it's best to specify the output format more rigidly in the Prompt.
Gemini 3.5
Gemini series models have always had good aesthetics, but there's one practical problem — instability.
Also, the running efficiency is relatively slow. Domestic models can complete this PPT task within 3 minutes, but using Gemini 3.5, a rough estimate is it had already been running for 10 minutes, and it was abnormally interrupted once.
If called within official tools, it would be more stable. The key issue is that Google's official tool, Google Antigravity, is also unusable.
Below is the generated PPT result.
If the task is abnormally interrupted, the task chain becomes disjointed, ultimately leading to worse consistency in the final product.
This is the final result, which took about 12 minutes because of the disconnection midway.
So Gemini 3.5's advantage leans more toward visual aesthetics and content expression, suitable for tasks with high demands on page quality. The weakness is end-to-end efficiency and chain stability.
For high-frequency, low-latency, production-grade Agent scenarios, this problem gets amplified.
GPT 5.4
Among mainstream foreign models, GPT may not have any particularly outstanding aspects, but it is relatively all-around. After all, GPT is the big brother of the model world.
The tool I used here is MonkeyCode, because this platform allows free use of GPT 5.4.
With the same prompt, here is the result:
This, like MiniMax, directly produced an HTML file. Clearly, it's not the PPT file we wanted.
Perhaps the wrong tool was selected. Switched back to the unified Agent tool Trae Work.
The result was mediocre, not particularly outstanding.
The cost was about 1.4 USD, which is roughly 9.5 RMB. It took about 10 minutes. Comparing this, it feels like aside from coding, for everyday AI use and Agent invocation, domestic models are entirely worth considering.
Model Time and Cost Comparison
| Model Name | Output Result | Time | Cost | Notes |
|---|---|---|---|---|
| Step 3.7 Flash | Successfully generated PPT file | ~5 min | ~1 RMB | Clear tool invocation, visible skills and searched websites, overall production-grade |
| DeepSeek V4-Pro | Successfully generated PPT file | ~5 min | ~0.5 RMB | Good content organization and presentation, more vibrant color scheme |
| MiniMax M3 | Generated HTML presentation | ~7 min | ~0.7 RMB | Good information organization and visual expression, but didn't call PPT tool as expected |
| Gemini 3.5 | Successfully generated PPT file | ~12 min | ~6 RMB | Good aesthetics, but experienced disconnection midway, average chain stability |
| GPT 5.4 | Final result mediocre | ~10 min | ~1.4 USD / ~9.5 RMB | First generated HTML, switched back to Trae Work with mediocre results, higher cost |
Information Scraping
Step-3.7-flash
Why test information scraping? Because this task requires the large model to call a browser tool, testing the model's ability to call a single tool for a single complex task. Browser information scraping requires the model to identify corresponding interface elements, such as where the likes are, where the comments are, find the corresponding elements, and then proceed with execution.
Prompt:
Go to Xiaohongshu and search for the hottest notes about Jimeng, select five, organize the note content, like count, and the first three comments into an HTML file, place it on the desktop, and name it "Note Organization".
This browser test task had relatively high consumption because at each step, the model had to think about what to do next, what element to click to obtain the corresponding data.
It consumed nearly 2 million tokens, costing about 0.9 yuan. The final result:
Successfully obtained data from Xiaohongshu.
MiniMax-M3
Using the same prompt, a test was conducted with MiniMax-M3. The data was clearly different from the previous one. The reason for the difference is the filtering criteria: MiniMax-M3 filtered by most likes, while Step-3.7-flash filtered by most comments.
The cost was around 1 yuan.
A minor issue with MiniMax-M3 is that it didn't open a browser to operate; it reached its conclusion using network searches within the Agent. But the generated result was acceptable.
DeepSeek-V4-Pro
DeepSeek-V4-Pro normally called the browser to obtain data.
The generated HTML result:
DeepSeek-V4-Pro used about 3.6 million tokens, costing around 0.5 yuan.
The testing ends here.
Model Time and Cost Comparison
| Model Name | Output Result | Time | Cost | Notes |
|---|---|---|---|---|
| Step 3.7 Flash | Successfully scraped Xiaohongshu data and generated HTML | Not recorded | ~0.9 RMB | Consumed nearly 2 million tokens, correctly obtained Xiaohongshu data |
| MiniMax M3 | Generated HTML data organization page | Not recorded | ~1 RMB | Did not open a browser to operate, mainly completed via Agent internal web search |
| DeepSeek V4-Pro | Successfully called browser and generated HTML | Not recorded | ~0.5 RMB | Used about 3.6 million tokens, normally called browser to obtain data |
Conclusion
The previous tests mainly examined the chain problem of an Agent task — from search → reading → summarization → code generation → to tool invocation, ultimately outputting PPT results and data displays. We focused on whether this process ran smoothly, as well as the end-to-end time and per-task cost.
If you only look at a single output, the gap might not be that exaggerated. But placed in a production environment, the differences get rapidly amplified. Because Agent tasks are about end-to-end results: can it run stably to completion, how long does it take to finish, how much does each invocation cost, and can the final file be used directly.
At least in "high-frequency, clear, verifiable" Agent tasks like these, the value of Flash-tier models begins to stand out. They don't pursue being first on every leaderboard, but find a more practical balance between speed, cost, and stability — and these are precisely the three dimensions that production-grade scenarios care about most.