Agent Benchmarks Don't Matter — Completion Rate, Speed, and Cost Do
Agent workflows amplify small reliability gaps into broken pipelines. A model that disconnects once or picks the wrong tool doesn't just produce a worse slide — it breaks the chain and forces a restart. For teams building automated multi-step agents, completion rate and per-task cost matter more than any benchmark number, and Chinese Flash-tier models are hitting a price-performance point that GPT and Gemini aren't matching yet.
Five models were given identical prompts inside the Trae Work Agent framework: research short-video platforms and produce a PPT, then scrape Xiaohongshu for trending posts and compile an HTML report. Step 3.7 Flash completed both tasks reliably in about 5 minutes for roughly 1 yuan, with transparent tool-call logging. DeepSeek V4-Pro matched that speed at half the cost and produced better-organized content. MiniMax M3 ignored the PPT tool and generated an HTML presentation instead — visually decent but the wrong format — and handled scraping via internal search rather than opening a browser. Gemini 3.5 took 12 minutes, disconnected mid-task, and cost 6 yuan. GPT 5.4 ran 10 minutes, cost 9.5 yuan, and delivered mediocre output after an initial tool-selection mistake.
Information scraping exposed deeper differences. Step 3.7 Flash burned nearly 2 million tokens navigating Xiaohongshu's DOM to extract likes and comments correctly. DeepSeek V4-Pro used 3.6 million tokens but cost only 0.5 yuan. MiniMax M3 sidestepped the browser entirely and relied on internal search, producing acceptable but differently-sourced results.
The takeaway is not that one model wins. Flash-tier models carve out a distinct niche: they trade benchmark-topping scores for the speed, cost, and completion reliability that production Agent pipelines actually demand.
Leaderboard rankings and launch-event demos measure a model's peak capability on isolated prompts; Agent tasks measure whether a model can chain multiple tool calls without derailing — and the two don't correlate well.
Tool-selection errors are a silent failure mode. MiniMax M3 and GPT 5.4 both produced usable output in the wrong format, which in a production pipeline means a downstream system breaks or a human has to intervene.
Cost differences compound fast: running GPT 5.4 at 9.5 yuan per task versus DeepSeek V4-Pro at 0.5 yuan means a 19x cost multiplier for worse results, which is unsustainable at any scale.
Gemini's instability inside a third-party Agent framework suggests that model quality is partly a function of the runtime environment; a model that runs well in a first-party sandbox may fall apart when orchestrated by external tooling.
MiniMax M3's decision to skip the browser and use internal search for scraping is a pragmatic shortcut that produced acceptable results — it raises the question of whether full browser automation is always worth the token cost.