跪拜 Guibai
← All articles
JavaScript

A Two-Week Browser Agent Run: The Model Was Fine, the Page Wasn't

By 西安小哥 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Teams building browser automation will keep buying better models and still get fake success logs if they ignore the page. The cheap, model-agnostic fixes – a stable element reference protocol and mandatory read-back rules – are where the real gains sit, and frontend engineers own half of that problem.

Summary

An internal expense form with 17 fields took an agent 14 minutes and failed on a custom dropdown; a human did it in 40 seconds. That gap isn't about model vision. The DOM structure of a div-wrapped select with a hidden select is the real obstacle, and no prompt rewiring fixes it.

Screenshot-based computer use fails on coordinate drift, high token costs, and unreliable state inference. DOM routes exist: accessibility trees collapse when semantics are missing, injected reference IDs become stale, and raw HTML is too noisy. Big vendors are moving to structured interfaces, MCP servers, or in-page scripting, all of which push determinism into the tool layer.

After two weeks, the author kept three practices: read back evidence after every click, never retry a failed click, and read APIs instead of reading the UI. None depends on model strength. The article also argues frontend code quality is half of agent reliability: adding data-testid and aria-label to legacy components cut script failure rates by more than half without touching the model. The final case study shows a model treating a popup's appearance as a completed bookmark; a hardcoded terminal-state assertion per step is the only cure.

Takeaways
— A pure visual screenshot approach costs thousands of tokens per image and fails on coordinate drift across viewport sizes.
— An accessibility tree is only useful if page authors wrote semantics; a div-wrapped button surfaces as unnamed text.
— Injecting temporary reference IDs eliminates coordinates but requires re-tagging each round and a rescan when IDs go stale.
— Raw HTML is too large for direct model input; it must be trimmed to interactive tags.
— MCP and browser modifications move determinism into the tool layer, not the model's perception.
— Read back evidence after every click: verify the input field is empty or a text appears, rather than trusting the click happened.
— Never retry a missed click; a second click can trigger a double-click or toggle an action like bookmarking into an undo.
— Reading information from an API endpoint is ten times cheaper and less ambiguous than reading it from a screenshot.
— Adding data-testid and aria-label to legacy components cut automation failure rates by more than half without changing the model.
— A model mistook a popup appearing for a completed bookmark, illustrating the trap of treating an intermediate state as a terminal state.
— Terminal-state assertions must be written by humans, not the model, because the model would pick the easiest condition to satisfy.
Conclusions

Agent reliability is largely a frontend code quality problem; pages that lack addressable interfaces will fail no matter how strong the model is.

Determinism belongs in the tool layer, not the model's perception; the three effective rules all live outside the model's capability.

Writing assertions by hand is a specific counter to agent self-reporting; any agent that defines its own success criteria will pick the path of least resistance.

Concepts & terms
Terminal-state assertion
A condition that must hold on the page after an action completes, used to distinguish actual success from an intermediate state. For example, after a bookmark action, the confirm button should no longer exist.
Accessibility tree (a11y tree)
A browser-generated structure exposing elements with roles, names, and states. It is cleaner and smaller than raw HTML but only works if page authors have written semantic markup.
MCP (Model Context Protocol)
A standard for exposing tool capabilities to models. A browser can act as a server that provides atomic actions like navigation, clicking, and execution, keeping deterministic behavior in the tool layer.
From the discussion
Featured comments
组合搭配函数

Not retrying after a single missed click is the most counter-intuitive yet most practical rule — clicking a toggle-type action a second time undoes it, favoriting becomes unfavoriting, and the Agent still thinks it succeeded. I felt the same building browser-side tools myself: if you nail element references and read-back evidence, even a rough model runs stably. And on that bit about the frontend shouldering half the blame — after adding data-testid and aria-label, the failure rate dropped by more than half with not one word of the model changed. That data is very convincing. A question: when you set the review standard for new components, do stable identifiers have to be on interactive components only, or also on purely presentational containers?

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗