How AI Agents Actually Drive a Browser: Accessibility Trees, Not Pixels
Understanding that agents drive browsers through accessibility trees — not pixels or raw DOM — explains why semantic HTML suddenly matters for AI reachability, why pure-text models can still automate the web, and where the real token costs hide in multi-step browser tasks.
Most AI agent CLIs default to a text-tree approach: the browser's built-in accessibility tree is pruned, numbered, and serialized into a compact YAML snapshot. The LLM reads that snapshot and issues tool calls referencing stable `ref=eNN` IDs, which Playwright resolves back to real DOM elements. Screenshot-and-coordinate mode is reserved for canvas, games, and visually-dependent targets where the tree is empty.
The loop is snapshot → select ref → operate → auto-return new snapshot. Old snapshots are collapsed to placeholders to keep token costs from ballooning across multi-step tasks. Browser tools are isolated in a sub-agent so the main agent's system prompt stays cacheable, and the MCP server starts lazily, reusing one browser instance per CLI session.
Visual fallback splits by provider: Anthropic gets inline images; other providers get a text caption generated by a borrowed vision model. Pure text models like DeepSeek work fine with the tree alone, making browser use viable without a multimodal model.
Browser automation for AI didn't require new infrastructure; it piggybacks on the accessibility tree that browsers have maintained for screen readers for over a decade.
The ref-numbering scheme is a clever indirection: the LLM operates on stable logical IDs while Playwright handles the messy DOM reality underneath, including detecting when an element has disappeared.
Sub-agent isolation isn't just architectural hygiene — it's a token-cost optimization. Keeping browser tools out of the main agent prevents system-prompt churn that would break prefix caching and raise per-request costs.
The split between Anthropic's inline-image path and the caption path for other providers reveals a real API fragmentation problem: most LLM APIs still can't handle images in tool results, forcing awkward workarounds.
Semantic HTML is now an AI-integration concern, not just an accessibility one. A div-based button without ARIA is invisible to an LLM driving the browser through the accessibility tree.
Token costs in browser use come primarily from snapshot size and accumulation, not from screenshots. A single dense page snapshot can be thousands of tokens, and multi-step tasks multiply that unless old snapshots are aggressively pruned.