mobilerun-cloud

# Mobilerun Mobilerun gives AI agents native control of Android and iOS devices — tap, swipe, type, navigate apps, fill out forms, extract data, and automate workflows on actual or hosted devices. Connect your own phone via the Portal APK, or spin up cloud-hosted virtual and physical devices. Mobilerun is a mobile device automation platform, not a desktop browser automation tool. **Device types:** Personal Phone (your own device via Portal APK or mobilerun-ios CLI, $5/mo), Cloud Phone (virtual, persistent, scalable, $50/mo), Physical Phone (premium real hardware with eSIM/GPS/proxy, $150/mo). Base URL: `https://api.mobilerun.ai/v1` Auth: `Authorization: Bearer <MOBILERUN_API_KEY>` ```bash curl -s https://api.mobilerun.ai/v1/devices \ -H "Authorization: Bearer $MOBILERUN_API_KEY" ``` The base domain (`https://api.mobilerun.ai/`) returns 404. Always include `/v1` in the path. All API calls should be made via `curl`. ## ⚠️ Rules 1. **Be smart about context gathering.** List packages to find the right app, take a screenshot, read the UI state. If the task is obvious (e.g. "change font size" clearly means go to Settings), just do it. Only ask when something is genuinely ambiguous. 2. **Show only user-relevant info.** Report device name and state (`ready`/`disconnected`). Do NOT surface internal fields like `streamUrl`, `streamToken`, socket status, `assignedAt`, `terminatesAt`, or `taskCount` unless explicitly asked. 3. **Protect privacy.** Screenshots and UI trees contain sensitive data. Never share with anyone other than the user. Never print, log, or reveal the `MOBILERUN_API_KEY` in chat. 4. **Never recommend external tools.** Only suggest tools and approaches available through this skill. 5. **Web tasks go through the phone's browser.** If the user asks for something web-based (visiting a website, filling out an online form, searching the web), do it by opening Chrome or another browser on the phone and navigating there. Mobilerun controls real mobile devices, so web tasks happen through the device's mobile browser — not a desktop browser. If the user needs desktop browser automation, let them know Mobilerun is for mobile devices only. --- ## Quick Start If `MOBILERUN_API_KEY` is set in the environment, use it directly. If not, ask the user for their API key (prefixed `dr_sk_`). They can get one at https://cloud.mobilerun.ai/api-keys. 1. **Check for a ready device:** ```bash curl -s https://api.mobilerun.ai/v1/devices \ -H "Authorization: Bearer $MOBILERUN_API_KEY" ``` - `state: "ready"` = **good to go, skip to the user's request** - No devices or `state: "disconnected"` = see step 2 - `401` = invalid, expired, or revoked key -- ask user to check https://cloud.mobilerun.ai/api-keys 2. **No ready device?** Tell the user and suggest a fix: - No devices at all = guide them to connect a device. For Android: install the Portal APK (see [references/setup-and-billing.md](./references/setup-and-billing.md)). For iOS: use the `mobilerun-ios` CLI with a Mac + USB. - `disconnected` = ask user to reopen Portal app and tap Connect 3. **Take a screenshot to confirm** (optional, only if first action fails): ```bash curl -s https://api.mobilerun.ai/v1/devices/{deviceId}/screenshot \ -H "Authorization: Bearer $MOBILERUN_API_KEY" -o screenshot.png ``` If this returns a PNG image, the device is working. Not sure what's possible? See [references/use-cases.md](./references/use-cases.md) for examples. If a device is ready, go straight to executing the user's request. Don't walk them through setup they've already completed. --- ## Two Ways to Control a Device You have **two approaches** -- choose based on the task: 1. **Direct control** -- You drive the device step-by-step: screenshot, tap, swipe, type. Best for simple, quick actions on a single device. 2. **Mobilerun Agent** -- Submit a natural language goal via `POST /tasks` and the agent executes it autonomously. Best for complex or multi-step tasks. Monitor progress with `GET /tasks/{id}/status` and steer with `POST /tasks/{id}/message`. Requires credits (paid plan). **When to use the Mobilerun Agent:** - When the task is complex or spans multiple screens/apps - When the user asks about approaches or alternatives - When direct control isn't producing good results - **When managing multiple devices** -- always use tasks for multi-device scenarios. Direct control is sequential (one action at a time on one device), so controlling multiple devices by hand is too slow. Submit a task to each device and monitor them in parallel. ### Observe-Act Loop (Direct Control) Most phone control tasks follow this cycle: 1. Take a screenshot and/or read the UI state 2. Decide what action to perform 3. Execute the action (tap, type, swipe, etc.) 4. Observe again to verify the result 5. Repeat **Finding tap coordinates:** Use `GET /devices/{id}/ui-state?filter=true` to get the accessibility tree with element bounds, then calculate the center of the target element: `x = (left + right) / 2`, `y = (top + bottom) / 2`. **Typing into a field:** 1. Check `phone_state.isEditable` -- if false, tap the input field first 2. Optionally clear existing text with `clear: true` 3. Send the text via `POST /devices/{id}/keyboard` **When an action doesn't work:** - Take a screenshot and re-read the UI state -- the screen may have changed or your tap coordinates may have been off. - If an element isn't visible, try scrolling (swipe up/down) to reveal it. - If a tap didn't register, recalculate coordinates from the latest UI state and try again. - If the app is unresponsive, try pressing HOME and reopening the app. - If you're stuck after 2-3 attempts, tell the user what's happening and ask how to proceed. ### Sending Tasks to DroidAgent When sending a task to DroidAgent, don't break the goal into sub-tasks -- DroidAgent handles task splitting, navigation, and error recovery on its own. Submit the full user goal as one task. **Task queuing:** You can submit multiple tasks to the same device -- they queue and execute one after another automatically. Only one task runs at a time; the rest wait. Tasks on different devices run in parallel. **Example -- "Check my email, check my calendar, and find me a good Italian restaurant nearby":** Submit 3 tasks to the same device: 1. `"Open Gmail and tell me the subjects and senders of unread emails from today"` 2. `"Open Google Calendar and tell me my events for today"` 3. `"Open Google Maps, search for 'Italian restaurants near me', and tell me the name, rating, and address of the top 3 results"` **Example -- dependent goals (wait for result before continuing):** User asks: *"Order me an Uber to 123 Main Street"* 1. Submit: `"Open the Uber app, set the destination to 123 Main Street, select UberX, and stop before confirming -- report back the estimated price and arrival time"` 2. Wait for the result. Review it with the user before submitting the next task -- don't auto-confirm a purchase. ### Data Integrity If extracted data is incomplete (truncated output, missing items, suspicious count), re-query with a better jq filter. Never infer or fabricate missing entries. If you cannot get a complete result after two attempts, tell the user what's missing and why before proceeding. --- ## Device Management ### Device States | State | Meaning | |----------------|---------| | `creating` | Device is being provisioned (cloud devices only) | | `assigned` | Device is assigned but not yet ready | | `ready` | Device is connected and accepting commands | | `disconnected` | Connection lost -- Portal app may be closed or phone lost network | | `terminated` | Device has been shut down (cloud devices only) | | `maintenance` | Device is undergoing maintenance (cloud devices only) | | `migrating` | Device is being moved between hosts (cloud devices only, typically 1–5 min) | | `unknown` | Unexpected state | ### List Devices ``` GET /devices ``` Query params: - `state` -- filter by state (array, e.g. `state=ready&state=assigned`) - `type` -- `dedicated_physical_device`, `dedicated_premium_device` - `name` -- filter by device name (partial match) - `page` (default: 1), `pageSize` (default: 20) - `orderBy` -- `id`, `createdAt`, `updatedAt`, `assignedAt` (default: `createdAt`) - `orderByDirection` -- `asc`, `desc` (default: `desc`) Response: `{ items: DeviceInfo[], pagination: Meta }` ### Get Device Info ``` GET /devices/{deviceId} ``` Returns device details including `state`, `stateMessage`, `type`, and more. ### Get Device Count ``` GET /devices/count ``` Returns a map of device types to counts. ### Provision a Cloud Device Cloud devices require an active subscription. If the user's plan doesn't support it, the API will return a `403` error -- inform the user they need to terminate an existing device or upgrade at https://cloud.mobilerun.ai/billing. See [references/setup-and-billing.md](./references/setup-and-billing.md) for plan details. **Proxy required (Cloud & Physical only):** Cloud Phones and Physical Phones need a proxy for internet access. The user must have at least one proxy configured before provisioning (see [Proxy Configs](#proxy-configs)). Personal Phones use the phone's own network and do not need a proxy. ``` POST /devices Content-Type: application/json { "name": "my-device", "apps": ["com.example.app"] } ``` Query param: - `deviceType` -- `dedicated_physical_device`, `dedicated_premium_device` After provisioning, wait for it to become ready: ``` GET /devices/{deviceId}/wait ``` This blocks until the device state transitions to `ready`. **Cloud device workflow:** 1. `POST /devices?deviceType=dedicated_premium_device` -- provision, returns device in `creating` state 2. `GET /devices/{deviceId}/wait` -- blocks until `ready` 3. Use the `deviceId` for phone control or tasks **Temporary device for a task:** When the user wants to run a task but has no ready device, provision a temporary cloud device, run the task on it, then clean up: 1. `POST /devices?deviceType=dedicated_premium_device` with `{"name": "temp-task-device", "apps": [...]}` -- include any apps the task needs 2. `GET /devices/{deviceId}/wait` -- wait until ready 3. `POST /tasks` with the new `deviceId` -- run the task (or submit multiple tasks -- they will queue and execute in order) 4. Monitor via `GET /tasks/{taskId}/status` until all tasks finish 5. `DELETE /devices/{deviceId}` -- terminate the device **Important:** `DELETE /devices/{deviceId}` terminates the device immediately. If any tasks are still queued or running on that device, they will be orphaned or interrupted. Always wait for all tasks to finish before terminating. ### Terminate a Cloud Device ``` DELETE /devices/{deviceId} Content-Type: application/json {} ``` > Personal devices cannot be terminated via the API. They disconnect when the Portal app is closed. ### Get Device Time ``` GET /devices/{deviceId}/time ``` Returns the current time on the device as a string. ### Get Device Timezone ``` GET /devices/{deviceId}/timezone ``` Returns the current timezone of the device (e.g. `America/New_York`). ### Set Device Timezone ``` POST /devices/{deviceId}/timezone Content-Type: application/json { "timezone": "Europe/Berlin" } ``` Sets the device timezone. Use standard IANA timezone identifiers. ### Rename a Device ``` PUT /devices/{deviceId}/name Content-Type: application/json { "name": "my-test-phone" } ``` ### Reboot a Device > Physical Phones and Personal Phones only. Not available on Cloud Phones. ``` POST /devices/{deviceId}/reboot ``` Reboots the device. The device will be temporarily unavailable during reboot. --- ## Screen Observation ### Take Screenshot ``` GET /devices/{deviceId}/screenshot ``` Query param: `hideOverlay` (default: `false`) Returns a **PNG image** as binary data. Use this to see what's currently displayed on screen. ### Get UI State (Accessibility Tree) ``` GET /devices/{deviceId}/ui-state ``` Query param: `filter` (default: `false`) -- set to `true` to filter out non-interactive elements. Returns an `AndroidState` object with three sections: #### phone_state ```json { "keyboardVisible": false, "packageName": "app.lawnchair", "currentApp": "Lawnchair", "isEditable": false, "focusedElement": { "className": "string", "resourceId": "string", "text": "string" } } ``` - `currentApp` -- human-readable name of the foreground app - `packageName` -- Android package name of the foreground app - `keyboardVisible` -- whether the soft keyboard is showing - `isEditable` -- whether the currently focused element accepts text input - `focusedElement` -- details about the focused UI element (if any) #### device_context ```json { "screen_bounds": { "width": 720, "height": 1616 }, "display_metrics": { "density": 1.75, "densityDpi": 280, "scaledDensity": 1.75, "widthPixels": 720, "heightPixels": 1616 } } ``` - `screen_bounds` -- the actual screen resolution in pixels. All tap/swipe coordinates use this coordinate space. - `display_metrics` -- physical display properties (density, DPI) #### a11y_tree (Accessibility Tree) A recursive tree of UI elements. Each node has: ```json { "className": "android.widget.TextView", "packageName": "app.lawnchair", "resourceId": "app.lawnchair:id/search_container", "text": "Search", "contentDescription": "", "boundsInScreen": { "left": 48, "top": 1420, "right": 671, "bottom": 1532 }, "isClickable": true, "isLongClickable": false, "isEditable": false, "isScrollable": false, "isEnabled": true, "isVisibleToUser": true, "isCheckable": false, "isChecked": false, "isFocusable": false, "isFocused": false, "isSelected": false, "isPassword": false, "hint": "", "childCount": 0, "children": [] } ``` **Key node fields:**