A Two-Week Browser Agent Run: The Model Was Fine, the Page Wasn't
Teams building browser automation will keep buying better models and still get fake success logs if they ignore the page. The cheap, model-agnostic fixes – a stable element reference protocol and mandatory read-back rules – are where the real gains sit, and frontend engineers own half of that problem.
An internal expense form with 17 fields took an agent 14 minutes and failed on a custom dropdown; a human did it in 40 seconds. That gap isn't about model vision. The DOM structure of a div-wrapped select with a hidden select is the real obstacle, and no prompt rewiring fixes it.
Screenshot-based computer use fails on coordinate drift, high token costs, and unreliable state inference. DOM routes exist: accessibility trees collapse when semantics are missing, injected reference IDs become stale, and raw HTML is too noisy. Big vendors are moving to structured interfaces, MCP servers, or in-page scripting, all of which push determinism into the tool layer.
After two weeks, the author kept three practices: read back evidence after every click, never retry a failed click, and read APIs instead of reading the UI. None depends on model strength. The article also argues frontend code quality is half of agent reliability: adding data-testid and aria-label to legacy components cut script failure rates by more than half without touching the model. The final case study shows a model treating a popup's appearance as a completed bookmark; a hardcoded terminal-state assertion per step is the only cure.
Agent reliability is largely a frontend code quality problem; pages that lack addressable interfaces will fail no matter how strong the model is.
Determinism belongs in the tool layer, not the model's perception; the three effective rules all live outside the model's capability.
Writing assertions by hand is a specific counter to agent self-reporting; any agent that defines its own success criteria will pick the path of least resistance.
Not retrying after a single missed click is the most counter-intuitive yet most practical rule — clicking a toggle-type action a second time undoes it, favoriting becomes unfavoriting, and the Agent still thinks it succeeded. I felt the same building browser-side tools myself: if you nail element references and read-back evidence, even a rough model runs stably. And on that bit about the frontend shouldering half the blame — after adding data-testid and aria-label, the failure rate dropped by more than half with not one word of the model changed. That data is very convincing. A question: when you set the review standard for new components, do stable identifiers have to be on interactive components only, or also on purely presentational containers?