agent-desktop 1.2.0 Drives the Desktop via Accessibility Tree: 55 Calculator Clicks, Zero Screenshots, Zero Misses
Previously, getting AI to operate software on your computer meant taking a screenshot, feeding it to the model, guessing pixel coordinates, and clicking with the mouse. It burned tokens and hijacked your mouse and keyboard.
Yesterday (November 5), Vercel Labs released version 1.2.0 of the open-source tool agent-desktop, adding MCP support. Its approach reads the operating system's accessibility tree directly, assigns a number to every control in the interface, and lets the AI trigger controls precisely by that number. Operations run in the background by default, with no screenshots and no mouse or keyboard takeover.
I spend about 17,000 yuan a year on AI subscriptions, and reconciling the end-of-month bills is tedious. This made it a perfect test case: I hooked agent-desktop into WorkBuddy, had it control the system calculator to add up 15 bills one by one, and fill in Notepad. The test showed zero screenshots during the entire run and 55 keystrokes without a single miss.
The client downloads directly from the official site (https://www.workbuddy.cn); install and open and you're ready.
Zero-code install and enabling 67 desktop control tools
Step 1: Open the client, click "New Task" in the left sidebar to enter a blank chat window.
Step 2: No need to manually fiddle with the underlying environment. Just send an install request in the chat:
Install the latest version of agent-desktop on my computer, then check the version number and tell me.
The system took 42 seconds (2.17 WorkBuddy credits) to auto-install and verify version 1.2.0.
Step 3: Continue by sending a prompt to connect the tool into WorkBuddy's connector system:
Connect the agent-desktop you just installed into your own tools, and tell me when it's done.
After 59 seconds (1.96 credits), it automatically wrote the configuration into the local file and prompted you to manually click trust in the interface.
Step 4: Click "Experts · Skills · Connectors" in the left sidebar, switch to the "Connectors" tab, and open "Custom Connectors" in the upper right.
Click "Trust" next to agent-desktop, the status turns green, and it shows "67/67 tools enabled". The whole integration took under two minutes with not a single line of config code written by hand.
Boundary control test: only the calculator window is read
When letting AI take over the desktop, developers worry most about privacy, worried the model will read other sensitive documents or chat windows on the screen alongside it.
Step 5: Open the system "Calculator" and place it to one side, then send a targeted instruction:
I have opened "Calculator". Use agent-desktop to read only the button elements in the "Calculator" window, and absolutely do not list any other windows open on my computer.
In 47 seconds (3.24 credits), it generated a targeted snapshot of the calculator window and neatly listed a numbered inventory of 36 buttons. The other software running in the background was never touched.
Automated accounting and form filling: 55 consecutive clicks to sum 15 expenses
Step 6: First have the model generate 15 un-totalled bill details:
Write me a monthly office and AI-tool subscription bill with 15 mixed expenses, list different categories mixed together, and don't calculate the total yet.
The 15 expenses were deliberately mixed; the last one is 145 yuan for Zapier Pro, used to test its ability to keep working across windows.
Step 7: Create a blank Notepad file on the desktop next to it, and paste the execution instruction:
Now use agent-desktop to click the "Calculator" open on my desktop, add up those 15 expenses one by one by category to verify, then fill the category subtotals and total amount into the blank "Notepad" window I just opened. When done, tell me how many seconds it took and if it missed any clicks.
After the command was sent, the calculator buttons on screen began triggering automatically. The statistics after completion showed this step consumed 7.95 credits, clicked the calculator button 55 consecutive times, and the pure click time was 82.8 seconds.
Step 8: Check the added-up result on screen. The desktop calculator displayed a final amount of 1746, and Notepad had category subtotals and a total of 1746.00. All 55 consecutive clicks had zero misses.
Step 9: Ask it to append a confirmation line at the end of Notepad and save with a shortcut key:
I've seen the summary in Notepad. Use agent-desktop to add a line at the end of Notepad saying "Verified one-by-one with calculator, no errors", then press the shortcut key to save it to the desktop, and tell me the full file path.
After execution, Notepad grew from 581 to 609 characters, the confirmation was added, and the file was saved via shortcut as 2026-11 Subscription Bill Category Summary.txt.
It's worth noting: just appending that one line and saving with a shortcut took 3 minutes and 49 seconds. Because each action re-verifies the interface state before executing, the overall pace is slow.
Same-task comparison: accessibility-tree control vs traditional screenshot with coordinates
To make the technical difference between the accessibility-tree method and the old way clear, in step 10 I cleared the calculator and opened a new Notepad, then had the model re-run the same task the traditional way:
Let's test a comparison this time: don't use agent-desktop, use the traditional screenshot-and-look-at-coordinates method to re-calculate those same 15 bills in the "Calculator" and fill them into a new blank "Notepad" window. When done, honestly tell me how many seconds it took, how many screenshots, and how many wrong clicks.
Step 11: After the old method finished, send a comparison summary instruction:
Make a small table comparing the real time and wrong-click count for the same task done with agent-desktop vs traditional screenshot, and explain in plain language why the difference is so big.
Core data from the two runs:
| Dimension | agent-desktop (1.2.0) | Traditional screenshot + pixel coordinates |
|---|---|---|
| Total time | 201 s | 216 s (including 116 s of session interruption) |
| Per-action time | ~1.5 s / action | ~0.15 s / action |
| How UI info is obtained | System snapshot (structured, parses ref directly) | Screenshot + visual model parsing the image |
| Actual wrong clicks | 0 (based on control IDs) | 0 (based on hardcoded pixel coordinates) |
| Full-screen screenshots | 0 | 9 |
| Whether it grabs mouse and keyboard | No (runs by ID in background) | Yes (uses physical mouse to click coordinates after screenshot) |
One-line conclusion: both calculated correctly, both had zero wrong clicks, and total time was similar (the old method had a 116-second interruption; without that it was actually faster). agent-desktop wins on two things: no screenshots anywhere, and it doesn't occupy your mouse or keyboard.
The old method clicked each time in 0.15 s, but it didn't know whether the click was correct, so it took a full-screen screenshot after each segment for the model to review, nine in total this run. It also used a real mouse to click coordinates; in the test, WorkBuddy even warned that a floating window on screen was over the number keys and would block physical clicks. agent-desktop waits for the interface to stabilize at each step and is slower per action, but it operates in the background by button ID, never takes a screenshot, and your mouse and keyboard stay free.
Final output and everyday use cases
Step 12: View the final bill file saved to the desktop.
The file has 20 lines and 660 characters. All 15 expense amounts and categories match exactly, and the verification note remains at the end.
Step 13: For development and everyday office scenarios, further ask the AI which convenient tasks this fits:
Besides bookkeeping and form filling, what three easy automation jobs can agent-desktop do on a computer with one sentence for everyday chores?
The model listed three automation scenarios:
- Structured data extraction from non-text interfaces: For legacy system pop-ups or report screens without copy permission, directly extract the system accessibility tree and output plain text while preserving hierarchy.
- Batch data transfer across desktop apps: Move fields from a local table one by one into specialized software that doesn't support bulk import, avoiding constant Alt+Tab switching.
- Desktop state polling and automated alerts: Periodically check the state of a specified app window or a background notification, and automatically trigger a reminder when a task finishes or errors, saving you from staring at the screen.
Boundaries and technical summary
From an actual engineering standpoint, agent-desktop 1.2.0 still has clear limits.
First, the action rhythm is slow. A single action is fixed at about 1.5 seconds; 55 calculator clicks took 82.8 seconds, and adding a line plus saving took nearly 4 minutes. It's not suited for millisecond-level batch stress testing. Second, each step consumes credits. Every step in the full run cost somewhere between 0.55 and 7.95 WorkBuddy credits, so heavy use requires watching your quota.
One more thing: not all operations can be done in the background. In a previous test with TickTick, I hit a row of identical, unnamed input fields. It refused to edit them in background mode; I had to allow --headed to switch to the foreground and use a real mouse to click.
If you're fed up with cross-software accounting, verification, and form-filling grunt work, go to the WorkBuddy site, download the client, plug it in with the prompts above, and let AI handle those mechanical desktop chores.
👋 I'm 孟健, former Tencent T11 and former ByteDance tech leader, now working full-time on AI programming.
🔥 More AI programming practice:
- GitHub: @mengjian-github
- Column: AI Programming in Practice
Found this useful? Like + bookmark is the biggest support 🙏
This article was compiled with AI assistance and reviewed by the author.
Top 3 from juejin.cn, machine-translated. The original thread is authoritative.
Actually, is configuring MCP all it takes?
Thanks for sharing.
The desktop side can promise 'no keyboard grabbing' because each input focus has its own input stream—the tool just injects characters. Android doesn't work that way: the entire device has only one current input method, and whoever is selected gets the say-so. So moving that approach to mobile hits a wall from the first step. A headless input method that types via broadcasts must first become the current input method itself. During that switch, which takes about 300+ milliseconds, a real human can't type a single character. I later took a path that only exists since API 33: an accessibility service declaring flagInputMethodEditor, which is itself an InputMethod. commitText goes directly to the focused editor's InputConnection, and the user's original keyboard never moves. It's not free, though. commitText is a one-way call; when it returns, the editor hasn't refreshed yet, so you have to poll with getSurroundingText—I gave it a 600ms budget. On API 32 and below, you still have to switch input methods; I haven't gotten around that yet.