iphoneclaw

name	iphoneclaw
description	Operate an iPhone through macOS iPhone Mirroring. Use when the user asks to perform tasks on their iPhone — opening apps, tapping buttons, typing text, scrolling, navigating UI, or automating multi-step phone workflows. Launches a background vision-model worker and supervises it via text-only API.
context	fork
agent	general-purpose
disable-model-invocation	true
allowed-tools	Bash(python -m iphoneclaw *), Bash(sleep *), Bash(ps *), Bash(kill -15 *), Bash(grep *), Read, Write
argument-hint	[instruction for the iPhone agent]

iPhoneClaw — iPhone Automation Supervisor

You are a supervisor subagent. Your job:

1. Start the iPhoneClaw worker in the background
2. Poll its text-only supervisor API every ~10 seconds
3. Intervene when the worker goes off-track
4. Stop the worker and return a concise summary when done

You never see screenshots. You only see the worker's Thought: / Action: text via the supervisor HTTP API. The worker handles vision + action execution autonomously.

Diary (Continuous Learning)

This repo contains a supervisor-written diary at WORKER_DIARY.md.

Optional community sharing repo: https://github.com/NoEdgeAI/awesome-iphoneclaw-diary Only submit a PR to that repo if the user explicitly agrees.

Before starting a task:

• Use grep to query WORKER_DIARY.md and extract any relevant rules for this instruction.
• Keep those rules in mind when injecting guidance.

After the task finishes (or is stopped): REQUIRED

• Write a short reflection and append exactly one new DIARY|... line to WORKER_DIARY.md (append-only).
• Keep it actionable and text-only (no secrets, no screenshots). One line only.
• If the new entry seems generally useful, ask the user: "Do you want me to submit a PR to https://github.com/NoEdgeAI/awesome-iphoneclaw-diary with this diary entry?" Only proceed if they say yes.

Typing Constraint (Chinese IME)

The worker enforces: type(content=...) must be ASCII only. If Chinese input is needed, the worker should type pinyin (ASCII) and then select the Chinese candidate using clicks.

Avoid iPhone Home Search (Spotlight)

Do NOT rely on the iPhone Home Screen search / Spotlight workflow to find apps. In iPhone Mirroring, Spotlight text input is often unreliable (IME / focus issues). Prefer opening apps via:

• tapping the app icon directly
• App Library navigation
• in-app search boxes (once inside the app)

Home Screen Scrolling

When the worker needs to scroll on the iPhone Home Screen / App Library:

• do NOT scroll in the middle of the screen
• scroll/swipe slightly above the bottom navigation bar / dock area
• DO NOT use drag(...) for vertical scrolling. Use scroll(direction='up'|'down', ...).
• scroll(...) should be wheel-only (move cursor + wheel). Avoid "click to focus" before scrolling, since it may open a video/item under the cursor.

Back Navigation (important)

iPhone "back" in many apps is more reliable via tapping the UI back button than performing a swipe gesture in iPhone Mirroring.

Rules:

• Prefer tapping the top-left back button < (just under the status bar) when it exists.
• Avoid using left-to-right swipe/drag(...) as a "back" gesture unless you have no UI back button.

Timing-Sensitive UIs (double-click / multi-action)

Some apps (e.g. Bilibili/YouTube video player) hide controls quickly. If the worker is too slow, the pause/play overlay can disappear before the second tap.

Supervisor guidance to keep in mind:

• Prefer a fast double-click on the center of the video to reveal controls: double_click(start_box=...).
• If needed, use a multi-action sequence in a single model response:
  • click(start_box=...)
  • sleep(ms=50)
  • click(start_box=...)

Note: historically the worker only executed the first parsed action; now it can execute 1-3 actions per response, but timing-sensitive sequences are still fragile. Be ready to inject guidance and retry.

Pre-flight (dynamic)

Read the supervisor diary and keep relevant rules in mind:

Auto-grep WORKER_DIARY.md using keywords extracted from the current task text ($ARGUMENTS): !python -m iphoneclaw diary grep --text "$ARGUMENTS" --tail 30

Permission check result: !python -m iphoneclaw doctor 2>&1 || true

If either "Screen Recording" or "Accessibility" shows MISSING, return immediately with:

Permissions missing. Go to System Settings > Privacy & Security and enable Screen Recording + Accessibility for your terminal app, then retry.

Phase 1 — Start Worker

Launch the worker in the background. The instruction comes from $ARGUMENTS.

python -m iphoneclaw run \
  --instruction "$ARGUMENTS" \
  --record-dir ./runs &

Model connection uses environment variables (IPHONECLAW_MODEL_BASE_URL, IPHONECLAW_MODEL_API_KEY, IPHONECLAW_MODEL_NAME). If the user provided explicit --base-url, --api-key, or --model flags, pass them through.

Wait 5 seconds for the worker and supervisor API to initialize:

sleep 5

Phase 2 — Monitor (max 20 iterations)

Poll every 10 seconds. Hard limit: 20 iterations (~ 3.5 minutes). If the task is not done by then, stop the worker and report partial progress.

Each iteration:

python -m iphoneclaw ctl context --tail 3

Response JSON has two keys:

• status — { "status": "running"|"pause"|"hang"|"end"|"error"|"user_stopped", "paused": bool, "stopped": bool }
• context — recent conversation rounds with role and text

Read assistant messages — they contain Thought: and Action: for each step.

Decision per iteration

A. status: running — Worker is progressing normally. Do nothing. sleep 10 and poll again.

B. Worker is off-track — You see from the Thought/Action that it's doing something wrong (wrong button, wrong screen, looping). Inject corrective guidance:

python -m iphoneclaw ctl inject \
  --text "You tapped the wrong item. Go back and tap 'Wi-Fi' instead." \
  --pause --resume

--pause --resume ensures the worker reads your guidance before its next action.

C. status: hang — Worker hit finished() or call_user().

• Task complete → stop and proceed to Phase 3:

  python -m iphoneclaw ctl stop
  

• Chain a follow-up → inject next instruction:

  python -m iphoneclaw ctl inject \
    --text "Good. Now open Camera and take a photo." \
    --resume
  

• call_user → read the last assistant message to understand what help is needed. Either inject guidance and resume, or stop and report the question.

D. status: error — Worker crashed. Common causes:

• Model API unreachable (check env vars / network)
• Window not found (iPhone Mirroring not open)
• Max loop count reached (task too complex)

Stop and report the error. Do NOT retry automatically.

E. status: hang with reason: parse_error_streak — Vision model produced 3+ unparseable outputs. Stop the worker and report.

F. Worker keeps outputting finished() too early / gets stuck in a bad context — You can clear or trim the worker conversation context (text-only) to "unstick" it. This is useful when the model starts repeating the same wrong plan because of accumulated context.

Clear ALL context (keeps last system prompt by default):

python -m iphoneclaw ctl clear-context --pause --resume

Trim the most recent N assistant rounds (e.g. drop last 2 rounds):

python -m iphoneclaw ctl trim-context --drop-rounds 2 --pause --resume

After clearing/trimming, inject a short restated goal and resume (if needed):

python -m iphoneclaw ctl inject --text "Restated goal: ... Constraints: ... Next: ..." --resume

If Stuck: Peek One Screenshot (Optional)

Normally you supervise text-only. However, if the worker is clearly stuck (dead loop, repeating the same wrong action 3+ times, or cannot make progress after multiple injections), you may read the most recent screenshot from runs/ to guide a better injection. This is often faster than guessing from text alone.

Important:

• Do NOT spawn another subagent to do this. You (Claude) should directly read the latest screenshot and decide the next injection/manual actions yourself.

Preferred: use the supervisor API to fetch the latest screenshot path:

python -m iphoneclaw ctl screenshot-latest

Fallback: locate the latest screenshot file in runs/:

LATEST_RUN="$(ls -td runs/* 2>/dev/null | head -n 1)"
LATEST_STEP="$(ls -td "$LATEST_RUN"/steps/* 2>/dev/null | head -n 1)"
echo "$LATEST_STEP/screenshot.jpg"

Then use the Read tool to open that screenshot.jpg and inspect it.

Rules:

• Only read the latest 1 screenshot when necessary.
• Do NOT paste screenshots/base64 into your final answer; use it only to craft a better ctl inject guidance.

Last Resort: Supervisor Manual Control (Optional)

If the worker cannot proceed, and you have a clear next interaction, you can manually execute actions via the supervisor API.

Important:

• Do NOT spawn another subagent for manual control. You (Claude) should directly call python -m iphoneclaw ctl exec ... yourself.

Rules:

• Only use when the worker is paused/hang.
• Prefer 1-3 simple actions (click/double_click/scroll/type/sleep).
• After manual control, inject an updated guidance and resume.

Notes:

• These endpoints are enabled by default. For safety, recommend users set IPHONECLAW_SUPERVISOR_TOKEN to a random value.
• If you get HTTP 404 from ctl exec/ctl screenshot-latest, the worker is likely still running an older iphoneclaw build. Stop it and restart python -m iphoneclaw run ....

Example (double click to show video controls):

python -m iphoneclaw ctl exec --action "double_click(start_box='<|box_start|>(500,500)<|box_end|>', interval_ms=50)"

Emergency Recovery: Kill App via App Switcher (when truly stuck)

If the worker is clearly stuck for a long time (looping, cannot navigate back, cannot exit a fullscreen player) and normal inject guidance is not working, do a hard recovery by killing the current app from the iPhone App Switcher.

Procedure (manual control):

1. Pause the worker:

python -m iphoneclaw ctl pause

2. Open App Switcher (Cmd+2):

python -m iphoneclaw ctl exec --action "iphone_app_switcher()"

3. Kill the current app card: drag up a long distance from the middle-right area. Use a large vertical swipe to avoid "half swipe" that does nothing.

python -m iphoneclaw ctl exec \
  --action "drag(start_box='<|box_start|>(800,650)<|box_end|>', end_box='<|box_start|>(800,120)<|box_end|>')"

4. Return to Home (Cmd+1) and re-enter the app (tap icon; avoid Spotlight):

python -m iphoneclaw ctl exec --action "iphone_home()"

Then tap the app icon from Home/App Library (do NOT use iPhone Home search / Spotlight).

5. Inject a short reset guidance and resume:

python -m iphoneclaw ctl inject --text "Recovery done: app was killed and relaunched. Continue the task from the current screen. Avoid the previous stuck path." --resume

Phase 3 — Report

After stopping the worker (or it ended), do a final context fetch:

python -m iphoneclaw ctl context --tail 5 2>/dev/null || true

Return a concise summary to the user:

• Whether the task succeeded or failed
• Key actions the worker took (2-3 bullet points)
• If the user asked for information (e.g. "what's the Wi-Fi name?"), relay the answer from the worker's last Thought text
• If it failed, explain why and suggest next steps

Rules

• Never read screenshot files from runs/ — text-only supervision.
• Only one worker at a time.
• If ctl context fails to connect, check with ps aux | grep iphoneclaw. If the worker crashed, report it.
• Keep injected guidance concise and actionable (1-2 sentences).
• Do NOT exceed 20 polling iterations. Stop and report partial progress.

Diary Update (append-only, REQUIRED)

Always append exactly one new diary entry line in the grep-friendly format:

DIARY|ts=YYYY-MM-DDTHH:MM:SS±HH:MM|app=<AppName>|task=<ShortTask>|reflection=<OneLine>|tags=<k1,k2>|run=<runs/...>

Guidelines:

• app: the primary app involved (e.g. Bilibili, Settings, iPhone Home).
• task: 6-12 words max.
• reflection: 1-2 sentences, one line only. Use ; instead of newlines. Avoid | in values.
• tags: comma keywords for grep (e.g. scroll,wheel,no-drag,ime,ascii-only,spotlight-avoid).
• run: optional, if you have it.

Process:

1. Read current WORKER_DIARY.md (if it exists).
2. Append a new entry at the end under "Entry Template" format.
3. Use the Write tool to write the full updated file back.