A Two-Week Browser Agent Run: The Model Was Fine, the Page Wasn't
theme: fancy
On the afternoon of November 10th, I had an Agent fill out an internal expense reimbursement form. Seventeen fields, it took fourteen minutes, and finally got stuck on a dropdown box—that box was a custom component, its real structure a div with a hidden select, clicks needed to land on that div. It tried seven methods, including stuffing a value into the hidden select and dispatching a change event, none worked. At the same time, I opened the page and filled it manually in 40 seconds.
This incident made me rethink: what exactly is hard about browser agents. Over the past six months, everyone has been discussing whether the model can see the screen clearly, whether coordinates are accurate. After running for two weeks, my conclusion is different: the difficulty lies in page uncertainty, and current solutions almost all go around it.
The screenshot route has a low ceiling
The pure visual approach (take a screenshot, let the model output click coordinates) was popularized by computer use, and many companies followed. Its advantage is generality: any interface can be seen, no adaptation needed.
There are three pieces of bad news, and they stack.
Coordinates themselves are unstable. The same button has different positions at different window widths and zoom levels, viewport 960 and 1280 are two coordinate systems. Coordinates given by the model occasionally drift to adjacent elements, and adjacent elements often happen to be delete.
Screenshots are expensive. One image enters context as thousands to tens of thousands of tokens, and a real task needs dozens. In two weeks, my most expensive consumption came from a pure visual task, costing enough money to manually complete it twenty times.
It lacks stable state. The visual model sees the submit button gray out and infers submission likely succeeded. That inference is wrong in many systems; graying might just be debouncing. A line of aria-disabled in the DOM makes it clear, yet it relies on guessing from pixels.
Three DOM routes, each with pitfalls
Current mainstream approach is not to only look at pixels but feed interactive elements to the model. Implementation has three variants.
Accessibility tree. The browser can export the a11y tree, role plus name plus state, much cleaner than HTML, smaller too. This is my recommended starting point. The pitfall: when page authors haven't written semantics, the exported tree is as unusable as screenshots. A button with div wrapping span appears as unnamed text in the a11y tree. My stuck expense form was exactly this case.
Tagging elements with references. The automation tool scans the page first, injects a temporary ID into each interactive element, the model only replies "click e761", the tool finds the element itself. This eliminates coordinates entirely, solving precision problems at once. The cost: re-tagging each round, IDs become invalid when the page moves, and the model occasionally references a vanished ID—at that point you must not guess, stop and rescan.
Stuffing HTML directly to the model. Convenient, but a modern page's HTML easily has hundreds of thousands of characters, mostly useless stylesheets and tracking attributes. To actually use it, you must trim first: keep only interactive tags, delete styles, scripts, and inline events.
Big companies are betting on a different direction
Looking around, the focus is shifting from "making the model more human" to "giving the model better interfaces."
One category is changing the browser itself. Several browsers for human-machine collaboration have appeared, where page content is opened to agents for structured reading rather than guessing through screenshots. This thinking is right: instead of optimizing eyes, let the interface speak clearly.
Another is the MCP route, standardizing tool capabilities. Browsers provide atomic actions like navigation, clicking, executing as servers, and the model only handles orchestration. Determinism thus lives in the tool layer, not the model's visual estimation.
Another kind bypasses the interface altogether, letting the model write scripts to execute in page context. This is the fastest and most fragile. Fast because one execution can replace dozens of clicks; fragile because it depends on selectors, and selectors are the easiest thing in frontend code to be refactored. I've suffered this: after a component library upgrade, a class name changed from submit-btn to form-action, my script silently failed for three days, and the success logs during that period were all fake.
Three small things that actually work
After two weeks, I kept just three changes, none fancy.
After every click, read back evidence. Don't trust "I clicked", trust "the input field is now empty" or "this text appears on the page". This rule basically eliminated false successes.
If a click misses once, don't retry. This is the most counterintuitive. If the agent clicks and it doesn't take effect, clicking again might trigger a double-click or close an already-open popup. For toggle actions like bookmarking or liking, a second click equals undo. My rule is hardcoded: if an action fails, stop and report, let the orchestration layer decide whether to change path.
If you can read an API, don't read the interface. Information like list progress is ten times cheaper to find in the DOM than from screenshots, and unambiguous. UI is left for parts that truly need human eyes.
By the way, none of these three relate to model capability. They are all process constraints, written in orchestration and tool layers. This is the point I find most worth saying: for the past six months we've gotten used to attributing agent problems to the model, but once actually running, you learn that engineering owes more debt than the model side.
A not-so-polite conclusion
The biggest takeaway from these two weeks: I adjusted my expectations of browser agents down a notch, and adjusted their usability up a notch.
Down because no matter how strong the model, it can't fix page uncertainty. An irregular frontend component can have the strongest model spend fourteen minutes there; this isn't something prompts can fix. Up because as long as you build determinism into the tool layer—reference tags, reading evidence, no retries—a crude model can completely do the job, and more stably than I imagined.
Conversely, if a team wants to do browser automation now, priority is not switching to a more expensive model, but adding a stable element reference protocol and a mandatory read-back acceptance rule. The former makes clicks discrete addressable actions, the latter separates looking successful from actually successful. The marginal benefit of these two things is much higher than switching models.
That reimbursement form that couldn't be filled, I later filled manually. But I copied the structure of that custom dropdown into the tool documentation, and now the same path on this system hasn't failed again.
The half of the blame frontend engineers should carry
As I write this, there's an unpleasant but necessary judgment: browser agent reliability is half determined by frontend code quality.
The same business, how it ends up depends entirely on whether the writer left addressable interfaces. Adding explicit role to buttons, writing stable IDs, avoiding hooking interactions on purely decorative divs: these things help human accessibility users and work for agents equally. Conversely, components that tie onMouseDown to svg, with real structure of three nested spans, let both humans and machines suffer.
Last month our team added data-testid and aria-label to a batch of old components. After that, the failure rate of the same automation script dropped by more than half, without changing a word of the model. This struck me more than any technical newsletter: testability in the agent era is conveniently usability.
My current practice is to make addressability an acceptance criterion in reviews: new interactive components must have a name, must have stable identifiers, not allowed to rely solely on class names. Sounds like making way for machines, but it also makes way for humans. Screen reader users have been waiting for exactly this.
A real failure review
Last week one left a deep impression. The task was archiving a batch of articles by tag. The agent clicked bookmark, a popup appeared, it replied "bookmark successful." Actually not successful—the popup still required checking a folder and clicking confirm; it took the popup appearing as completion.
This error is typical: not that it didn't see, but it treated an intermediate state as a terminal state. Visual approaches especially stumble on such multi-step popups because the screenshot looks "changed," the model thinks the task advanced. DOM judgment can directly ask: is that confirm button still there? If yes, the flow isn't complete.
I later added a "terminal state assertion" for each step: a condition that must hold on the page after that thing is done. Writing an assertion costs little, one line of text, but it's the only antidote to false success. An agent without terminal state assertions, its success rate report can only be read as fiction.
By the way, the model can't write the assertions itself. It would pick the easiest to satisfy. Assertions must be hardcoded by humans in the task description, or brought down from requirements docs. This is orchestration layer responsibility, shouldn't be delegated.
Top 1 from juejin.cn, machine-translated. The original thread is authoritative.
Not retrying after a single missed click is the most counter-intuitive yet most practical rule — clicking a toggle-type action a second time undoes it, favoriting becomes unfavoriting, and the Agent still thinks it succeeded. I felt the same building browser-side tools myself: if you nail element references and read-back evidence, even a rough model runs stably. And on that bit about the frontend shouldering half the blame — after adding data-testid and aria-label, the failure rate dropped by more than half with not one word of the model changed. That data is very convincing. A question: when you set the review standard for new components, do stable identifiers have to be on interactive components only, or also on purely presentational containers?