android-harness
Drive a real Android phone from this PC over ADB. Nothing is installed on the
phone.
When to Use
- The user asks for something to happen ON their phone: open an app, read the
screen, tap, type, swipe, navigate a flow.
- The task only exists in a mobile app (no web or desktop equivalent).
- The user wants to read a notification, an SMS, or an app's on-screen state.
Don't use for: anything achievable on the desktop or the web — those are
faster, cheaper and need no device. This is for phone-only work.
Usage
Pipe a Python program in; helpers are already in scope. One turn = one script =
one screenshot, so do several steps per invocation rather than one call per tap:
android-harness <<'PY'
launch("com.android.settings")
wait_stable()
obs = observe(rung=2)
print(obs.compact)
tap_text("Network & internet")
PY
Check the connection first if anything looks wrong: android-harness --doctor
The loop
observe(rung=1) — pixels + state. Cheap (~800 tokens). This is the default.
- Act:
tap_text(...), tap(x, y, frame), type_text(...), launch(pkg, uri).
wait_stable() — never sleep().
observe() again to verify. The capture is the ground truth.
Escalate with observe(rung=2) for numbered tappable elements, rung=3 for the
full tree, rung=4 for OCR. Every observation reports its cost in
obs.receipt — check it, and do not escalate without a reason.
Rules
- Prefer
launch(pkg, uri="...") deep links over tapping through menus: they
skip whole sequences and are immune to layout drift.
- Use
tap_text rather than raw coordinates when a label exists. It resolves
the label to its nearest clickable ancestor, which is usually a different
node than the one holding the text.
- Irreversible actions (pay, send, delete) are gated and will prompt the human.
Do not try to route around the gate.
- A black screen means the app set FLAG_SECURE. That is respected, not
bypassed: check
obs.capture_protected, then use the tree or ask the human.
type_text is ASCII-only on this version. Non-ASCII (Georgian, emoji) raises
a clear error — ask the human to type it. Reading non-ASCII works fine.
- Write helpers you wish existed into
agent-workspace/agent_helpers.py; they
are auto-loaded into every later script.
Connection is the human's job
The harness never enables debugging or authorises itself. If --doctor reports
the device is unauthorized, STOP and ask the user to tap "Allow" on the phone
(ticking "Always allow from this computer"). Never retry it in a loop.
Pitfalls
- Nothing reads your stdin, so this is safe to run from a harness with no TTY.
- Set
ANDROID_HARNESS_UNATTENDED=1 when no human is watching (an agent
tool call, cron, CI). Gated actions then deny immediately instead of waiting
out the approval timeout.
- Without that, a gated action prompts on the console and denies after 60s
if nobody answers. It never hangs indefinitely. A
GateDeniedError in an
unattended run is the gate working — relay it to the human, do NOT retry in
a loop and do NOT try to route around it.
--doctor first whenever behaviour looks strange; it names the cause rather
than leaving you guessing.
Verification
After any action, observe() again and confirm the screen changed as expected.
The capture is the ground truth — never assume a tap landed. Each run also
writes a self-contained HTML replay under runs/ showing what was seen and done.