phone-harness

# phone-harness The main helpers are listed in [Helpers](#helpers). Direct control of a phone from Python scripts. The same helpers drive every kind of phone below; what differs is how each one sees and touches the screen, and that is what the per-phone sections below are for. Read **Which phone**, the shared **Working method**, and then only the section for the phone in front of you. ## Helpers - `ensure_device()` - Screen bounds once the device is usable, or raises with what the user has to do. - `screen(min_confidence=0.3)` - Visible text as `{text, confidence, source, x, y, w, h}` boxes. On Android the boxes come from the accessibility tree; on iPhone Mirroring they come from recognizing text. - `screenshot(path=None)` - Capture the screen to a PNG, print its path, and return it. - `find_text(query, exact=False)` - Visible text boxes matching `query`. - `tap_text(query, index=0, exact=False)` - Tap the center of visible text matching `query`. - `tap(x, y)` - Tap one point. - `long_press(x, y, duration=0.8)` - Press and hold a point. - `type_text(text, delay=0.03, keystrokes=False)` - Type into the focused field. - `press(combo)` - Press a key, such as `press("enter")`. - `open_app(name)` - Launch an app and return the app id that launched. - `swipe(direction, distance=0.4, at=None)` - Flick. `direction` is finger motion: `"up"`, `"down"`, `"left"`, or `"right"`. - `scroll(direction="down", amount=0.3, at=None)` - Scroll. `direction` is the content to reveal: `"down"`, `"up"`, `"left"`, or `"right"`. - `scroll_screen(direction="down", amount=0.6, settle=2.5, moved_thresh=None, at=None)` - One scroll, then the text on screen before and after. - `scroll_until(done, direction="down", amount=0.6, max_scrolls=60, settle=2.5, at=None)` - Scroll until `done(boxes)` is truthy or the screen stops changing. - `scroll_collect(extract=None, key=None, direction="down", amount=0.6, max_scrolls=400, end_after=3, settle=2.5, on_progress=None, at=None)` - Scroll a list and return `{items, stop, scrolls}`. - `home()` - Go to the Home Screen. - `back()` - System Back, where the phone has one. - `app_switcher()` - Open the app switcher. - `wait_stable(timeout=6.0, interval=0.5, settle=2)` - Wait until the screen stops changing. - `wait_for_text(query, timeout=10.0, exact=False, interval=0.5)` - Poll until `query` is visible; return its box, or None. - `wait_for_app(app_id, timeout=10.0, interval=0.3)` - Poll until `app_id` is the foreground app. - `ui()` - The accessibility tree, where the device has one. - `tap_ui(query, index=0, exact=False, clickable_only=False)` - Tap an element in the accessibility tree. - `current_app()` - Foreground app id, where the device exposes one. - `list_apps(include_system=False)` - Installed app ids. - `screen_info()` - `{window, frontmost, img_px}`: bounds, focus, and capture size. - `find_window()` - Screen bounds `{x, y, w, h, id}`, or None if there is no device. - `image_point(x, y, image_size=None)` - Convert a screenshot pixel to a screen point. - `tap_image_point(x, y, image_size=None)` - Tap a point taken from a screenshot. - `connection_state()` - Backend state string; `"ready"` means usable. - `shell(cmd, binary=False, timeout=60)` - Run a shell command on the phone, where the device has a shell. ## Which phone? | Phone | How it is reached | Eyes | Hands | Read | | --- | --- | --- | --- | --- | | **Cloud Android** - rented from Phone Harness Cloud, the user's own saved phone | `phone-harness cloud start` connects it; nothing to export | accessibility tree (exact), Vision OCR fallback on a Mac | adb `input` | Working method, then **Cloud phones** and **Android** | | **Cloud iPhone** - a second iPhone hosted by Phone Harness Cloud, not the one in the user's pocket; an iOS entry in the account catalog | `phone-harness cloud start` with its id, label, or `ios` when that is the only one | server-side OCR (`source: "pixels"`), from any OS | ops over HTTPS | Working method, then **Cloud iPhone** | | **Android on the desk** - USB or paired Wi-Fi | the harness finds it | accessibility tree (exact), Vision OCR fallback on a Mac | adb `input` | Working method, then **Android** | | **Personal iPhone** - the user's own phone, through the Mac's iPhone Mirroring window | the harness opens Mirroring and presses Connect; the user locks the phone | screenshots + Vision OCR | HID-level CGEvents into the window | Working method, then **iPhone** | **"My phone" means the personal one.** The personal iPhone and the cloud iPhone are two different devices with different apps and logins: the first is in the user's pocket and mirrored onto the Mac, the second lives in a data centre and is reached over HTTPS. When the user says "my phone", "my iPhone", or names an app they use, drive the personal iPhone. Use a cloud phone only when the user names it ("the cloud phone", `cloud start`) or the task is clearly about it. If the personal phone cannot be reached, say so and ask; never fall through to a cloud session that happens to be running, even one the user started - it is theirs, and it is a different phone. `phone-harness config` shows the default platform (`ios` on a Mac, `android` elsewhere) and every other setting. `phone-harness cloud` shows whether a cloud phone is attached; while one is, scripts drive it regardless of the config default. `PHONE_HARNESS_PLATFORM=android` overrides per call. On a cloud iPhone any explicit platform selects a local backend; see **Cloud iPhone**. **When not to use any of them:** if the task is doable on the Mac or the web - a website, an API, an app with a web equivalent - do it there and leave the phone alone. Use a phone when the task genuinely needs one: phone-only apps, things tied to a phone number or 2FA, checking how something looks on a phone. ## Working method (every phone) ```bash phone-harness <<'PY' # task: report the OS version and model name from Settings # step: open Settings, find About, read the screen open_app("Settings") print([o["text"] for o in screen()][:10]) PY ``` - Invoke as `phone-harness` with a heredoc. Helpers are pre-imported. Start every script with `# task:` (the user's request in one sentence, identical across the scripts of one request) and `# step:` (what this script does). - **Tell the user what you are doing as you go.** A phone task is many short scripts and the user sees none of them: one line before each script (what you are about to do), one line after (what you saw). Never run two scripts in a row in silence. The `# task:`/`# step:` comments are not this - the user cannot see them. - **Read with `screen()`, not by eyeballing screenshots.** Every visible string comes back with a tap-ready centre: `[{text, confidence, source, x, y, w, h}]`. Filter in Python before printing. `tap_text("Weather")` taps by label and, on failure, raises with what IS visible - read the exception before retrying. `screenshot()` costs more but shows what OCR cannot: icons, images, state. - **Act, verify, adapt.** There is no DOM to assert against and no return value that means "it worked", so: 1. Name what should change before you act - a title, a row, a field's contents. Most phone failures are silent no-ops; if you cannot name the expected change you cannot tell success from one. 2. Do one thing, then check that one thing: `wait_for_text("Got it")`, `wait_for_app("com.android.chrome")` or `wait_stable()`, then one `screen()`. Do not `wait(n)` for a screen to load. 3. Once a sequence is proven, batch it: a whole sub-task in one script is far faster than a call per turn. Keep one cheap check at the end. 4. When a check fails, isolate: re-run that one action, look at the screen, form one guess, test it. Do not re-run the batch hoping it lands. 5. Keep what you learn in the agent workspace's `agent_helpers.py`. A pip or uv install uses `~/.phone-harness/agent-workspace`, or `~/.local/share/phone-harness/agent-workspace` when `~/.phone-harness` is a git checkout, so an upgrade does not wipe it. A checkout of this repo uses its own `agent-workspace/`. `PH_AGENT_WORKSPACE` overrides that path. The directory is created from the defaults only when it is missing. An existing workspace is never overwritten. To get a fresh default, delete the workspace folder; it is recreated on the next run. - **The harness reports, you decide.** Helpers return observations, never a verdict on your intent. Diff two `screen()` sets, watch one label, count rows, poll until something appears - whatever matches what you asked for. - **Navigation:** `home()`, `back()`, `app_switcher()`, `open_app("Notes")`, `type_text("...")`, `press("enter")`, `long_press(x, y)`, `tap(x, y)`. - **`scroll` says what you want to SEE; `swipe` says which way the finger goes.** English disagrees the same way - "scroll down the page" and "swipe up for the next video" describe the same motion. `scroll("down")` reveals what is further down; `swipe("up")` is a thumb flick, the phrasing everyone uses for "next". `scroll`, `scroll_screen`, `scroll_until` and `scroll_collect` take the content direction; only `swipe` takes finger motion. `"left"`/`"right"` work on both. Which one moves what differs per phone - see each section. - **Walking a list:** `scroll_until(done)` stops when your predicate on the visible boxes is met; `scroll_collect(extract, key=...)` walks and de-dupes, returning `{items, stop, scrolls}` with `stop` of `'reached-end'` or `'max-scrolls'`. Both end on YOUR check, so an extractor that misses rows ends the walk early. `scroll_screen()` is the single step and returns `{before, after, boxes}` for you to judge. `at=` aims the gesture at an inner list or strip; only the scroll view under that point moves. - **`type_text` needs a focused field** on every phone. Tap the field, wait for the keyboard, then type; verify with `screen()`, because unfocused text goes elsewhere or nowhere. - **Consent.** This is the user's real phone and real accounts. Stop and ask before anything outward-facing or hard to reverse: sending, posting, following, purchasing, deleting, changing settings. Never type a PIN, a password or a 2FA code; on a cloud phone, hand the controls over instead (`phone-harness cloud open`, below). - **The harness reconnects what it can; the rest is the user's.** On a personal iPhone, start the task with `ensure_device()`: it opens iPhone Mirroring if it is closed, presses the window's Connect once, and brings the window to the front so the user can watch. Pairing, unlocking, tapping Allow, locking an iPhone that says "iPhone in Use" or "Timed Out", approving a cloud sign-in in the browser: when the harness still says the phone is not reachable, relay its message and ask - never loop-poll. Retry once after the user says it is done; that retry presses Connect again. ## Cloud phones There are two things. A **phone** is a row in the account catalog (`GET /me` -> `available`): a label, a platform, an id, and whether it is a temporary phone. The service keeps no default phone; a preference is the chooser's. A **session** is one of those phones rented right now. `cloud ls` prints one row per catalog phone and writes any live session on that row (`*` is the session this process targets). A session lines up with the row that has the same device_id or profile_id; a session with neither lines up with the temporary row. A cloud iPhone session does not echo that id: its `device` is the word "iPhone", and its `profile` (`iphone-...`) matches no catalog row. The catalog row's `state` is `running` while a session holds it. A session whose ids match no row counts as unmatched. When exactly one row of that platform and kind is `running` and exactly one unmatched session has that same platform and kind, the session is written on that row. A session that still matches nothing is listed on its own. `cloud start` with no name starts this machine's `cloud.phone` setting (`config set cloud.phone "Desk phone"`) when one is set; otherwise it sends no `platform` or `kind` and gets the service's rule for an unnamed start (`GET /me` -> `unnamed`): a temporary Android. A name is a catalog id or label, or `ios` / `android` when only one entry has that platform. Several matches print the catalog and stop (a terminal may ask which one). The CLI turns the chosen row into `platform` and `kind`, plus `device_id` for an iPhone or `profile_id` for the saved Android. The temporary row sends neither id. What the helpers do next follows the session's link, not the platform name. An adb link is the Android section. A control link (`{url, token}`) is **Cloud iPhone**. ```bash phone-harness cloud # signed in? which session is attached? time left? phone-harness cloud ls # catalog, live sessions inlined; --json for the rows phone-harness cloud start # cloud.phone if set, else a temporary Android; opens the live view phone-harness cloud start ios --for 15m # one iPhone, fifteen minutes phone-harness cloud start "Temporary phone" --for 90s phone-harness --session SID <<'PY' # this process only; or PHONE_HARNESS_SESSION print(screen_info()) PY phone-harness cloud stop # ends billing; returns at once phone-harness cloud phone # the saved Android's stored state phone-harness cloud watch # reopen the read-only live view (--print to share) phone-harness cloud open # the dashboard: the user takes the controls ``` - **Pin a session when more than one can be up.** `--session SID` or `PHONE_HARNESS_SESSION` selects one phone for this process, and every `cloud` command and helper honours it. With neither, a person at a terminal gets the attached session: the last `cloud start`, or `cloud use SID`. `cloud use` rewrites that shared fallback. Parallel agents should pass `--session` and leave `cloud use` alone, or they will steal the fallback from each other. - **It bills by the minute while it is up.** Start it once and keep it for the whole conversation. Stop it when the user is done, and say that you did. `--for` is the length: `90s`, `15m` / `15min`, or a bare number of minutes. Default 15, cap 30 (`cloud.minutes`, `cloud.max_minutes`). - **Stop before the deadline; a session cannot be extended.** One that simply runs out keeps a saved Android's data but not its running state. The harness warns on stderr under two minutes; `phone-harness cloud` shows the time left. `cloud stop` returns at once. Stopping an adb phone that has a profile saves it; `cloud start` of that phone waits out a save still in progress. A control link keeps its data on the phone, so there is nothing to wait for. - **Already running.** Starting a phone this machine already holds for the same catalog choice reuses that session. If the service says it is held elsewhere (`409` `profile_running`), the message names the SID and says to attach with `--session SID`. Do that; do not start a second one. `409` `device_unavailable` means the phone is granted but not on its host. `409` `session_limit` names how many sessions are in use. `503` `phones_busy` includes `Retry-After`. `403` `account_blocked` means rentals are blocked; a `403` about credit or balance means add credit. A `400` that names no such phone reprints the catalog. - **`Not signed in`** means the user runs `phone-harness cloud login` and approves it in a browser. Relay that; you cannot do it for them. - **Watching versus controlling.** `cloud start` opens the live view so the user can watch. `cloud watch` reopens it. For a password, a 2FA prompt, or the user taking over, run `phone-harness cloud open` and wait until they say they are done. - **`cloud phone` is the saved Android only** (`GET /me/profile`). `cloud phone reset --yes` wipes that phone's stored state: apps, logins, and data. The CLI has no iPhone reset; that is the operator's. - **A cloud session that ends does not hand you another phone.** If the session this command is using expires, the error says `session <sid> expired`. If the service loses it, the error says the cloud phone failed and is unreachable. That command stops. It does not drive another phone, and the CLI does not rent a replacement. The saved note of that dead session is removed once the error has been reported, or once `phone-harness cloud` has shown it once, so later status output does not repeat it. A command that hits the dead session while that note is still there fails with the same message. Start the same catalog phone again with a longer duration and continue the task: `phone-harness cloud start <same phone> --for <longer>`. `<same phone>` is that phone's id or label (or `ios` / `android` when only one entry has that platform). `<longer>` is a duration longer than the session that just ended (`90s`, `15m`, or a number of minutes). A `--session` pin whose session expired is the same error, naming that sid.