跪拜 Guibai
← All articles
Artificial Intelligence · Testing · Open Source

ARTEMIS: Google's Open-Source Agent That Operates Android Phones Like a Human

By 狂师 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

This gives Western developers a working alternative to scripted mobile UI automation: describe a task in natural language and let the agent handle the taps, swipes, and verification on real hardware. The MCP integration means it drops directly into existing AI coding workflows, and the self-drawn UI fallback covers the blind spots that have long frustrated uiautomator-based testing.

Summary

ARTEMIS is Google's open-source AI agent that automates real Android devices from a single plain-language instruction. It finds UI controls through a three-layer strategy—accessibility hierarchy, coordinates, and OCR plus vision for self-drawn pages like Compose or Flutter—so it works even on interfaces that traditionally break scripted locators. The first run installs a local accessibility helper that never sends data off-device.

Two execution profiles handle different workloads: Flash (3–5 seconds per step) for deterministic tasks like installation or login, and Pro (15–40 seconds per step) which plans, verifies each action against the current UI tree and pixels, and does a final check against the original goal. History compression keeps long tasks from blowing up the context window. For developer workflows, a built-in MCP server connects to Claude Code and other AI IDEs with one command, and a Web console offers live screen casting with a task queue and replay.

Setup is two commands: clone and run `./start.sh`, which pulls ADB, scrcpy, FFmpeg, and Python dependencies. A Python SDK with Pydantic-typed results integrates into pytest and CI. Caveats: Android only for now, each step is a model call so token costs pile up, and apps without accessibility support fall back to less reliable coordinates and vision.

Takeaways
— ARTEMIS is Google-issued, Apache-2.0 licensed, and free to use.
— It uses a three-route control search: accessibility hierarchy first, coordinates second, and OCR plus vision for self-drawn pages like Canvas, Compose, and Flutter.
— Two profiles: Flash (3–5 seconds per step) for deterministic tasks, Pro (15–40 seconds per step) with planner-operator-checker architecture for complex or long tasks.
— History compression keeps context manageable for tasks with hundreds of steps.
— On Google Research's AndroidWorld benchmark it reports a 99%+ task completion rate, though the benchmark is self-scored.
— A native MCP server mounts into Claude Code, Codex, Antigravity, and Windsurf with one command.
— The Python SDK returns Pydantic-typed results and can be used in pytest and CI pipelines.
— Only Android is supported; iOS and on-device lightweight models are on the roadmap.
— Each step is a model call, so token cost and runtime are real constraints.
— Apps with poor accessibility support force a fallback to coordinates and vision, which reduces stability.
Conclusions

The benchmark number is officially self-scored: Google Research wrote the questions and Google submitted the answer, so treat the 99% as indicative rather than an independent third-party evaluation.

The three-layer control-discovery pattern is a practical answer to the long-standing pain point of self-drawn UIs, which have always been a blind spot for uiautomator-based tools.

The token cost per step makes plain-script regression suites the better choice for thousands of test cases; this agent is targeted at interactive and exploratory tasks, not bulk automation.

Concepts & terms
AndroidWorld
A benchmark from Google Research that scores AI agents on their ability to complete real multi-step tasks on real Android devices across more than 20 apps and 100+ tasks.
MCP Server
Model Context Protocol server, a standard interface that lets AI coding tools like Claude Code connect to external tools and services, allowing an agent to invoke them directly from a chat dialog.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗