agent-desktop 1.2.0 Drives the Desktop via Accessibility Tree: 55 Calculator Clicks, Zero Screenshots, Zero Misses
Because it reads the accessibility tree instead of screenshots, agent-desktop removes the token cost of image parsing and frees the user's mouse and keyboard while the agent works in the background. The price is a slower per-action pace and a per-step credit system, which makes it a fit for unattended desktop automation rather than millisecond-level interactions.
Agent-desktop 1.2.0, released by Vercel Labs on November 5, replaces the usual screenshot-and-pixel-coordinate approach with direct reads of the OS accessibility tree. Each UI control gets a number, and the AI triggers it by that number in the background, with no screenshots and no mouse or keyboard takeover.
A hands-on test chained agent-desktop into WorkBuddy: install and connect took under two minutes without writing config code, then the AI read only the Calculator window's 36 numbered buttons, generated 15 mixed bill items, added them up with 55 consecutive clicks in 82.8 seconds with zero errors, and filled a Notepad file saved via shortcut. A side-by-side comparison against the traditional screenshot method showed both runs were correct, but agent-desktop took zero full-screen screenshots and never touched the physical input devices, while the old method needed nine screenshots and occupied the mouse.
The trade-offs are explicit: each action takes about 1.5 seconds because the model re-checks the interface state before acting, every step consumes WorkBuddy credits, and controls that lack stable identifiers may require falling back to foreground mode with a real mouse.
The 1.5-second-per-action rhythm is the deliberate cost of verification: the model re-reads the interface state before every click, which is why it never misses but also why adding one line and saving took nearly four minutes.
Reliability in UI automation now depends on the quality of the accessibility tree, so applications that expose no names for their controls still force a fallback to headed mode and physical clicks.
This comparison positions MCP-based desktop control as a token-saver and user-convenience play, not a speed play; for tasks where a human is idle, the slower, non-intrusive approach is the practical win.
Credit-based metering per step means the economic cost of agentic desktop use scales with the number of UI actions, not just the complexity of the task.
The discussion centers on the portability of the accessibility-tree method to Android. The desktop's keyboard-free claim rests on per-focus input streams; Android's single current input method destroys that. A workaround exists only from API 33—an accessibility service that is itself an InputMethod, using commitText directly—though it requires getSurroundingText polling with a 600ms budget, and older versions still need a full input-method switch. The simple question of whether MCP configuration is all that's required stands unanswered against the detailed technical caveats.
Actually, is configuring MCP all it takes?
The desktop side can promise 'no keyboard grabbing' because each input focus has its own input stream—the tool just injects characters. Android doesn't work that way: the entire device has only one current input method, and whoever is selected gets the say-so. So moving that approach to mobile hits a wall from the first step. A headless input method that types via broadcasts must first become the current input method itself. During that switch, which takes about 300+ milliseconds, a real human can't type a single character. I later took a path that only exists since API 33: an accessibility service declaring flagInputMethodEditor, which is itself an InputMethod. commitText goes directly to the focused editor's InputConnection, and the user's original keyboard never moves. It's not free, though. commitText is a one-way call; when it returns, the editor hasn't refreshed yet, so you have to poll with getSurroundingText—I gave it a 600ms budget. On API 32 and below, you still have to switch input methods; I haven't gotten around that yet.