跪拜 Guibai
← Back to the summary

ARTEMIS: Google's Open-Source Agent That Operates Android Phones Like a Human

Title: Awesome! Google itself open-sources an AI automation artifact! Author: 狂师 Tags: Artificial Intelligence, Testing, Open Source

Let AI operate a phone like a human—look at the screen, swipe, tap, and do the whole thing from start to finish. Google has stepped in to do this, with a project called ARTEMIS. It was open-sourced in August, and in the three months since, it has racked up 17k+ stars.

Repetitive operations on phones—installation, registration, login, going through settings one by one—mean an Android tester taps tens of thousands of times a year. Previously, to avoid this, you either kept tapping manually or wrote a stack of scripts that fed the locator of every button to the machine.

ARTEMIS takes a different approach: you describe what you want in one plain sentence, and it handles the rest itself.

For this project, you only need to remember a few key facts.

What it does is simple, summed up in one sentence: You give a task in natural language; the AI Agent looks at the screen, finds controls, swipes and taps, completes the task on a real Android device, and brings back screenshots and logs.

AI operates a real device like a human

Official demo animation: a single command, it opens Google Maps, sets a driving route, computes total time, then switches to YouTube and plays a Coldplay song. Two apps, one sentence, no one touches the phone in between.

Official demo, one command completes a task across two apps

Its operation logic is similar to a human: look at the screen, think, tap.

It finds each control on the interface through three routes.

Priority goes to Android's accessibility hierarchy, directly fetching the control index. If that fails, it falls back to coordinates. For interfaces with no nodes in the control tree—like Canvas, Compose, Flutter—it uses OCR plus a vision model as a safety net. Anyone who has done mobile automation knows that self-drawn pages used to be a blind spot for scripts; uiautomatorviewer would show a blank.

The first time it runs a task on a device, it automatically installs an accessibility Helper on the phone. This helper only reads the screen layout locally and sends no data out. The service doesn't occupy the UiAutomation connection; if it crashes mid-run, it automatically falls back to UIAutomator2 and continues, with the failure noted in the task timeline.

Two modes, switch freely

Running tasks is divided into Flash and Pro, toggled with a single --profile command-line argument. Flash is for speed, Pro is for stability.

Flash is the fast tier: a single model runs a reactive look-and-do loop, 3–5 seconds per step. It's for deterministic operations like installation, login, and tapping through settings. Pro is the slow tier: 15–40 seconds per step. A Planner first produces a task plan with milestones; an Operator executes it, and before each action is sent out, it checks against the current UI tree and pixels to verify the target is still there. When done, a Checker does a final review against the original objective and outputs a written report.

Long tasks don't crash because of history compression. Completed steps are compressed into retrievable summaries; old screenshots are collapsed into visual summaries. Context doesn't grow infinitely with step count, so even tasks with hundreds of steps hold up.

There are also official benchmark numbers. Google Research's AndroidWorld benchmark covers 20+ apps and 100+ multi-step tasks, specifically testing an agent's ability to complete real tasks on real devices. ARTEMIS scored 99%+ completion rate. Since Google Research wrote the questions and Google also submitted the answers, take this number as an order of magnitude. whether it works on your own app, you'll have to verify hands-on.

AndroidWorld benchmark comparison

Test directly in your code editor

ARTEMIS comes with a native MCP Server, mounted into an AI IDE with one command.

uv run artemis mcp --install all

After installation, in Claude Code's dialog, you can directly say:

Build the latest changes into an APK, install it on the connected device, walk through login with a test account, verify there are no unexpected popups after login, and finally give me a screenshot of the page.

It handles the build, installation, operations, and brings back Logcat logs and screenshots. The project also includes a mobile testing guideline rules.md that teaches the AI Agent to explore first before acting, and to prefer dynamic locators before falling back to coordinates; it gets mounted into the IDE during installation.

Bug reproduction is particularly smooth in this scenario: send a user-reported operation path exactly as-is, let it run on a real device, and it brings back the scene exactly as it happened.

Official workflow, command from IDE to real-device execution

Web screencast, watch it work

If you don't want to open an editor, run ./start.sh and the browser will automatically open a Web console.

On the left is a real-device screen cast; in the middle is the live inference stream showing where it tapped, the coordinates, and each step's conclusion, all visible. The input box at the bottom issues tasks in natural language; the right panel manages the task queue and history replay. If a run fails and you want to see what happened, click replay.

Web console

Two commands to run

The barrier is just two steps: you need a real Android device with USB debugging enabled, or an emulator.

git clone https://github.com/google/artemis.git && cd artemis
./start.sh

start.sh automatically installs ADB, scrcpy, FFmpeg, and Python dependencies, then prompts you to mount the MCP service into your AI IDE. If you skip the browser, you can run from the command line directly:

uv run artemis run "Open system settings and check the current battery" --profile flash

To use it in an existing project, it provides a zero-runtime-dependency Python SDK that returns Pydantic-typed results. You can write assertions directly into pytest, and it can be wired into CI pipelines.

A few things to watch out for

Only Android is supported; iOS and on-device lightweight models are still on the roadmap.

Each step is a model call, so token cost and time are real factors. For regression suites with thousands of cases, running plain scripts is still cheaper. Conclusions carry probability; don't hand over a binary pass/fail quality gate to it wholesale. Also, it relies on accessibility services to read the screen; if your app strips accessibility support, it falls back to coordinates and vision, and stability degrades.

Closing

Whether you use the tool or not, the ideas are worth borrowing. The fast/slow two-tier execution strategy for matching different tasks, the anti-hallucination design of verifying targets before each action, and the history-compression long-task approach all apply to your own testing tools and processes.

Start with one or two repetitive phone operations, hand them to it—installation, login, checking popups—run a few times to see success rates. That's more useful than taking sides.

GitHub project address:

https://github.com/google/artemis