mobile-harness

name	mobile-harness
description	Direct iOS/Android device control via MobAI's DSL. Use when the user wants to automate, test, or interact with a real phone, simulator, or emulator. Requires MobAI to be running locally.

mobile-harness

Direct device control via MobAI's /dsl/execute. Each Python primitive (tap, type_text, swipe, ...) is one DSL step; agents compose primitives into reusable helpers.

For task-specific edits, use agent-workspace/agent_helpers.py and agent-workspace/domain-skills/. For setup, install, or connection problems, read install.md.

Two modes - same primitives

Immediate (outside run()): each primitive posts a 1-step script and returns the unwrapped action result. Use for exploration, REPL-style probes, and branching flows.

Batched (inside run(thunk)): each primitive returns a step dict; the whole thunk collects into one DSL script that posts as a single HTTP call. Use for known deterministic flows. MobAI's explore mode skips intermediate observes for free.

# Immediate
mobile-harness -c '
open_app("com.apple.Preferences")
wait_for(stable=True, timeout_ms=3000)
print(observe(include=["ui_tree"]))
'

# Batched - same primitives, one HTTP call
mobile-harness -c '
result = run(lambda: (
    open_app("com.apple.Preferences"),
    tap(Predicate(text_contains="General")),
    wait_for(Predicate(text_contains="About"), timeout_ms=3000),
    observe(include=["ui_tree"]),
))
print(result["step_results"][-1]["result"]["observations"]["native"]["ui_tree"])
'

• Invoke as mobile-harness - it's on $PATH. No cd, no uv run.
• All helpers, Predicate, Near, run, steps_of, StepFailed are pre-imported.
• Requires MobAI's HTTP API at http://127.0.0.1:8686/api/v1. Override with $MOBAI_URL.
• Pin a device with MOBAI_DEVICE=<id> or call set_default_device("<id>"). With one device connected, defaults work.

When to use which mode

• Use run(...) for any sequence longer than ~3 steps that's deterministic (login, search, navigate-and-extract). Fewer HTTP calls; one ExecutionResult; intermediate observes skipped by MobAI's explore mode; clean failure trace via StepFailed.step_result.debug.ui_tree.
• Use immediate calls when the next action depends on screen content you haven't observed yet, or when probing.
• Domain-skill flows ship as two functions: *_steps (composable) and the bare name (immediate). The _steps form emits primitives without an internal run() so it composes inside a larger batch. The bare form is run(my_flow_steps) for one-shot use. See agent-workspace/domain-skills/ios-settings/__init__.py for the pattern.

The gotcha

observe() returns two different shapes depending on mode. Mixing them up is the most common bug. The harness gives a clearer error if you key into the batched-shape keys on an immediate return, but .get() chains will silently return the default — so know the shape.

Immediate (obs = observe(...)): the per-context Observation, keys at top level.

• Right: obs["ui_tree"]
• Wrong: obs["observations"]["native"]["ui_tree"]

Batched (r = run(lambda: (..., observe(...)))): the full ExecutionResult.

• Right: r["step_results"][-1]["result"]["observations"]["native"]["ui_tree"]
• Wrong: r["ui_tree"]

Inside run(), primitives return step dicts, not results. Don't branch on their return values:

# BROKEN - observe() inside run returns a step dict, not the observation
def flow():
    state = observe(...)
    if "Search" in state.get("ui_tree", ""):  # always falsy
        tap(...)

# Right - branch in Python AFTER run() returns
r = run(lambda: (tap(...), type_text(...), observe(...)))
ui = r["step_results"][-1]["result"]["observations"]["native"]["ui_tree"]
if "Search" in ui:
    tap(...)

# Or use the DSL's screen-state branch inside the script
run(lambda: (
    tap(...),
    if_exists(Predicate(text="Allow"),
              then=steps_of(lambda: tap(Predicate(text="Allow")))),
    observe(...),
))

The inverse mistake — applying the batched shape to an immediate return — fails silently when written with .get() defaults:

# BROKEN - immediate observe() already returns the unwrapped observation
obs = observe(include=["ui_tree"])
ui = obs["observations"]["native"]["ui_tree"]                        # KeyError (with hint)
ui = obs.get("observations", {}).get("native", {}).get("ui_tree", "") # silently ""

# Right - keys are at the top level
ui = observe(include=["ui_tree"])["ui_tree"]
ui = ui_tree()  # convenience wrapper that returns the string directly

The loop

observe → plan → act via primitives → verify → repeat

1. Observe first. observe(include=["ui_tree"]) or screenshot(). The DSL reference says: never ask the user for screen state - observe it.
2. Plan in Python, not in DSL JSON. Branching, loops, parsing, and data shaping happen in Python helpers.
3. Act with primitives. One primitive = one DSL step = one HTTP call.
4. Verify with wait_for(stable=True) then observe() before assuming the action worked.
5. Use assert_screen_changed() to catch tap-missed / no-op cases. Pattern: observe(...) → action → delay(...) → assert_screen_changed(). Do NOT observe again before the assert - it resets the baseline.

open_app activates by default, fresh=True resets

open_app(bundle_id) matches the OS primitive - if the app is already running, it brings the existing process to the foreground in whatever state it was last in. Settings remembers the last-visited pane, App Store remembers your last tab, etc. This is not "open fresh."

For deterministic flows, pass fresh=True. MobAI kills the existing process first so the app starts at its root state. Every domain-skill flow should start this way unless it's intentionally continuing from current state.

open_app("com.apple.Preferences", fresh=True)  # always starts at Settings root
open_app("com.apple.Preferences")              # could land on whatever pane was last visited

Follow open_app(fresh=True) with wait_for(stable=True, timeout_ms=3000) before tapping anything - fresh launches take ~1-2 seconds to render their initial view.

Predicates beat coordinates

The DSL ships rich predicates - that's the reason the harness uses /dsl/execute rather than the granular HTTP API. Always prefer predicates over (x, y).

# good
tap(Predicate(text_contains="Continue", type="button"))
tap(Predicate(text_contains="Email", type="input", near=Near(text_contains="Login", direction="below")))
tap(Predicate(parent_of=Predicate(text="Submit")))

# fallback only when the element isn't in the UI tree (custom-rendered, OCR-only)
tap(x=180, y=420)

Common rules from the device-automation reference:

• Prefer text_contains over exact text - UI text shifts with state and locale.
• Add type alongside text when several element types share the text.
• iOS apps with sparse UI trees (React Native, Flutter, system dialogs): use observe(include=["ocr"]) and tap by OCR coords.
• Use scroll(direction="down", to=Predicate(...)) - semantic scroll - to find off-screen elements. Not swipe.

Siri shortcut beats UI navigation (iOS)

Before tapping through 5 screens to reach a feature, check if the app exposes a SiriKit intent via observe(include=["installed_apps"]) and call siri("Open my cart in Amazon"). Faster, more reliable, survives redesigns.

Search domain-skills first

Before inventing a flow for a known app, check agent-workspace/domain-skills/ for the bundle id or app name:

rg --files agent-workspace/domain-skills
rg -n "appstore|com.apple" agent-workspace/domain-skills

Always contribute back

If you learned anything non-obvious about how an app works, file it under agent-workspace/domain-skills/<app-slug>/ before you finish. The harness gets better only because agents file what they learn. If figuring something out cost a few attempts, the next run should not pay the same tax.

Examples worth saving:

• A predicate that disambiguates a duplicate-text screen (type:button + bounds_hint:bottom_half instead of plain text_contains).
• A siri prompt that replaces a 6-tap navigation.
• A React Native / Flutter screen where the UI tree is empty and OCR is required.
• A flow that needs an extra wait_for(stable=True) after a transition.
• An if_exists for a permission or onboarding popup that intermittently appears.
• An app-specific bundle id, deep-link URL scheme, or Siri intent name.

What a domain-skill should capture

The durable shape of the app - the map, not the diary:

• Bundle id, deep-link schemes, registered Siri intents.
• Stable predicates for key elements.
• App structure - tabs, modal patterns, where state lives.
• Quirks unique to this app (sparse UI tree, custom renderer, dialogs that hijack input).
• Waits and the reason they're needed.
• Predicates that don't work and why.

Do not write

• Raw pixel coordinates. They break across devices, orientations, and OS versions.
• Run narration of the specific task you just completed.
• Secrets, session tokens, user-specific credentials. domain-skills/ is shared and public.

What actually works

• Observe with ui_tree, not screenshots. UI tree is faster, smaller, and what predicates match against. Use screenshots for visual verification (layout, colors, images) only.
• Two screenshot primitives - pick the right one.
  • screenshot() → cheap, downscaled JPEG, file path. HTTP only (not batchable). Use for LLM visual analysis.
  • save_screenshot() → full-quality PNG, file path. Dual-mode (works inside run()). Use for reports / debugging / sharing.
  • observe(include=["screenshot"]) also returns a file path (MobAI saves server-side). Fine to use when you want the screenshot together with the ui_tree in one call.
• Batch deterministic sequences with run(...). Any flow longer than ~3 steps that doesn't branch on intermediate screen content should be wrapped - fewer HTTP calls, cleaner trace, and MobAI's explore mode skips intermediate observes for free. For branching, keep primitives immediate or use if_exists inside the batch.
• wait_for(stable=True, timeout_ms=3000) then observe(...) is the standard verify pattern after any action that triggers a transition or animation.
• Predicates over coordinates, always. Coordinates only when the element is OCR-only or when MobAI's tree is genuinely empty.
• scroll(to=Predicate(...)) is the right way to find off-screen elements. swipe up is a physical gesture; scroll down is semantic ("look further down").
• siri(...) before manual navigation on iOS, when the app supports it.
• For data collection from infinite-scrolling views: scroll to load a batch, then observe(only_visible=False) to grab everything in one go. Never use only_visible=False while interacting - off-screen elements cannot be tapped.

Design constraints

• One primitive = one DSL step = one HTTP call. Compose in Python, not in DSL JSON.
• Predicate and Near are dataclasses - they serialize via .to_dict(). Plain dicts also accepted by every primitive.
• No daemon. MobAI's HTTP API is stateless, so every call is a fresh request.
• Core helpers stay short. Task-specific additions go in agent-workspace/agent_helpers.py.
• No retries framework, session manager, config system, or logging framework. The DSL has on_fail and wait_for; that's enough.

Gotchas

• Single device assumption. With one device connected, primitives auto-default. With multiple, set MOBAI_DEVICE or call set_default_device(id).
• only_visible=False is for collection, not interaction. Off-screen elements can be returned but not tapped.
• iOS has no hardware back. Use swipe(direction="right", from_x=0) from the left edge, or rely on in-app back buttons. Android has navigate("back") and press_key("back").
• open_app failure usually means crash. Don't retry. Use metrics_start(capture_logs=True) or open_app(debug=True) for debug builds, then diagnose.
• assert_screen_changed: don't observe between the action and the assert - it resets the baseline.
• Sparse UI tree (React Native / Flutter / system dialogs on iOS): fall back to observe(include=["ocr"]) and tap by returned coords.

Web context (webviews)

For tapping inside webviews / hybrid apps, drop into a web_context() block. Predicates use css_selector / xpath instead of native fields.

select_web_context(url_contains="checkout.example.com")
with web_context():
    type_text("[e-mail]", predicate=Predicate(css_selector="input#email"))
    tap(Predicate(css_selector="button.continue"))
    title = execute_js("return document.title")
    dom = observe(include=["dom"])

• iOS physical devices only (no simulators); Android physical or emulator.
• Always try native automation first. Drop to web context only when DOM access is genuinely needed (deep webview state, JS evaluation, complex CSS selectors that don't map to native predicates).
• navigate(url=...) works inside web_context() to drive page navigation.
• execute_js(...) returns the JS expression result (use async_=True when awaiting promises).

Tool call shape

mobile-harness -c '
# any python. helpers, Predicate, Near pre-imported.
'

For batched flows, wrap in run(thunk) (recommended) or call run(steps=[...]) with raw step dicts. execute([...]) is the lowest-level escape hatch (returns the raw ExecutionResult without StepFailed unwrapping). mobai_post("/devices/X/...") is the raw HTTP escape hatch.