mobile-app-flows

# Omi Mobile App — Flows & Exploration This skill teaches you the Omi Flutter mobile app's navigation structure, screen architecture, and widget patterns. Use it when developing features (to understand how the app works), fixing bugs (to navigate to the affected screen), or verifying changes (to confirm your code works in the live app). > **Verifying changes (agents and contributors):** this skill drives a *live* > app on an emulator/device and assumes a manually-authenticated session > against a real or local backend — it is an exploration and physical-device > tool. The canonical fast verification path is the seeded local lane with > synthetic auth: `make mobile-verify ARGS="fast --paths <changed-file>"` — > no device, no OAuth, loopback fixtures. See > [`scripts/dev-harness/MOBILE_VERIFY.md`](../../scripts/dev-harness/MOBILE_VERIFY.md) > and [`MOBILE_SESSIONS.md`](../../scripts/dev-harness/MOBILE_SESSIONS.md). > Physical-device evidence remains a separately reported lane (SCA-491). ## How to Explore the App You can interact with the running app via `agent-flutter` — a CLI that taps widgets, reads the widget tree, and captures screenshots through Flutter's Marionette debug protocol. ### Setup ```bash # 1. Emulator must be running adb devices # should show emulator-5554 # If not: sg kvm -c "$ANDROID_HOME/emulator/emulator -avd omi-dev -no-window -gpu swiftshader_indirect -no-audio -no-boot-anim &" # 2. Set system language to English (REQUIRED — non-English IME breaks text input) adb shell "settings put system system_locales en-US" adb shell "setprop persist.sys.locale en-US" # 3. App must be running in debug mode with flutter run stdout captured cd app && flutter run -d emulator-5554 --flavor dev > /tmp/omi-flutter.log 2>&1 & # Wait for "VM Service" line to appear in the log # 4. Connect agent-flutter (AGENT_FLUTTER_LOG must point to flutter run stdout, NOT logcat) AGENT_FLUTTER_LOG=/tmp/omi-flutter.log agent-flutter connect agent-flutter snapshot -i --json # see what's on screen ``` **Prerequisites:** - AVD name: `omi-dev` (check: `$ANDROID_HOME/emulator/emulator -list-avds`) - KVM access required: user must be in `kvm` group (`sg kvm -c "..."` if not in current session) - App package: `com.friend.ios.dev` (dev flavor) - **System language must be English** — non-English IME breaks `fill` commands - **App must be authenticated and connected to the correct backend** (local, dev, or prod — depends on the task) - Marionette already integrated: `marionette_flutter: ^0.3.0` in pubspec.yaml ### Setup (iOS physical device) Full iOS physical-device playbook (setup, driving, verification, troubleshooting): [`IOS_DEVICE_TESTING.md`](./IOS_DEVICE_TESTING.md). Verified 2026-07-11 on an iPhone XR (iOS 18.7.9), app 1.0.543, prod-flavor debug build, agent-flutter CLI + marionette MCP. iOS Simulator has no BLE — use a physical device for anything beyond onboarding/UI checks (see also iOS Simulator Known Limitations below). ```bash # 1. Get the physical device id flutter devices # 2. Run in debug mode with stdout captured — prod flavor (dev flavor can fail codesigning on a # fresh worktree: "provisioning profile doesn't support the App Groups capability") cd app && flutter run -d <device-id> --flavor prod > /tmp/omi-flutter.log 2>&1 & # Wait for "A Dart VM Service ... is available" in the log # 3. Connect agent-flutter — auto-detects the ws URI from the log AGENT_FLUTTER_LOG=/tmp/omi-flutter.log agent-flutter connect agent-flutter snapshot -i --json ``` **Gotchas:** - **Fresh git worktrees need real prod config seeded from the primary checkout** before building the `prod` flavor: copy `app/.env`, `app/lib/firebase_options_prod.dart`, `app/ios/Config/Prod/GoogleService-Info.plist`, `app/lib/env/prod_env.g.dart`. The `test.sh` bootstrap seeds dev-placeholder versions that point at the wrong Firebase project. - **adb-backed commands do NOT work on iOS**: `back`, `press x y`, `dismiss`, `text --press`, `text --fill`. Use in-app back buttons (find the top-left `IconButton` ref) and marionette ref presses only. - **`fill @ref` can silently no-op on keyless `TextField`s on iOS** — it reports success but the controller stays empty. Verify via the marionette MCP `get_interactive_elements` (shows the `TextEditingController` contents). Durable fix: add a `ValueKey` to the field, hot reload (sheet state survives), then enter text by key. - **`snapshot` labels are empty with `marionette_flutter` 0.3.0** on iOS. Primary orientation/assertion tool is `agent-flutter text` (semantic text dump) — assert outcomes by text presence (e.g. the copy snackbar text). Target elements by type + bounds from `snapshot -i`. - `agent-flutter screenshot` output path must be under `/tmp`. - Check behavior via the run log: `grep -iE 'exception|error' /tmp/omi-flutter.log` after each flow. A `PlatformException` 4001 (Intercom push token, notifications not granted) is benign. - iOS terminates the debug connection if the app is backgrounded/locked too long ("The OS has terminated the Flutter debug connection for being inactive") — keep the device unlocked; reconnecting requires relaunching `flutter run`. - General key guidance (same as Android): prefer `find key "name"`; when a control can't be targeted, add a `ValueKey` in source + hot reload rather than fighting coordinates. ### Commands | Command | Purpose | Example | |---------|---------|---------| | `snapshot -i --json` | See all interactive widgets with refs, types, bounds | `agent-flutter snapshot -i --json` | | `press @ref` | Tap a widget by ref | `agent-flutter press @e3` | | `press x y` | Tap by coordinates (ADB input tap) | `agent-flutter press 540 1200` | | `press @ref --adb` | Tap by ref using ADB (for stale refs) | `agent-flutter press @e3 --adb` | | `dismiss` | Dismiss system dialogs (location, permissions) | `agent-flutter dismiss` | | `find type X press` | Find widget by type and tap | `agent-flutter find type button press` | | `find text "X" press` | Find by visible text and tap | `agent-flutter find text "Settings" press` | | `find type X --index N press` | Tap Nth match (0-indexed) | `agent-flutter find type switch --index 0 press` | | `fill @ref "text"` | Type into text field | `agent-flutter fill @e7 "search"` | | `scroll down/up` | Scroll current view | `agent-flutter scroll down` | | `back` | Android back button | `agent-flutter back` | | `screenshot PATH` | Capture current screen | `agent-flutter screenshot /tmp/screen.png` | **Key rules:** - Refs go stale frequently (Flutter rebuilds widget tree aggressively) — always re-snapshot before every interaction, not just after mutations. - `find type X` is more stable than hardcoded `@ref` numbers. - `AGENT_FLUTTER_LOG` must point to `flutter run` stdout (not logcat). - After hot restart: `disconnect` → wait 3s → `connect`. - Widget text labels are often null — use `type`, `flutterType`, or `bounds` to identify. - `back`, `press x y`, `dismiss`, and the `--adb`/`--press`/`--fill` text flags are ADB-backed — Android only. On a physical iOS device use marionette ref-based commands instead (see Setup (iOS physical device) above). ### Recovery ```bash # "No isolate with Marionette" → bring app to foreground + reconnect adb -s emulator-5554 shell am start -n com.friend.ios.dev/com.friend.ios.MainActivity agent-flutter disconnect && agent-flutter connect # Unhealthy widget tree → hot restart kill -SIGUSR2 $(pgrep -f "flutter_tools.*run" | head -1) sleep 3 && agent-flutter disconnect && agent-flutter connect ``` ## App Navigation Architecture ### Screen Map ``` Onboarding (wrapper.dart) — step wizard ├── 0: Auth (auth.dart) — Google/Apple sign-in ├── 1: AI Consent ├── 2: Name (name_widget.dart) ├── 3: Primary Language (primary_language_widget.dart) ├── 4: Found Omi (found_omi_widget.dart) ├── 5: Permissions (permissions_widget.dart) ├── 6: User Review (user_review_page.dart) ├── 9: Speech Profile (speech_profile_widget.dart) ├── 10: Knowledge Graph (knowledge_graph_step.dart) └── 11: Complete (complete_screen.dart) → Home Home (home/page.dart) — main app after auth, 4-slot bottom nav ├── ["Ask Omi" input bar] → Chat (chat/page.dart) — full-width bar above bottom nav, not a tab │ ├── Message history, "Ask anything" field, AI responses │ └── AI-message action row: Copy ("✨ Message copied to clipboard" snackbar), thumbs up, thumbs down, Share ├── [mic in the bar] → Chat with voice auto-start ├── [battery/record widget, top left] (battery_info_widget.dart) │ ├── Device connected → battery pill → Connected Device (home/device.dart); phone icon → Phone Calls │ └── No device → Connect → Connect Device page; Record pill → Conversation Capturing (conversation_capturing/page.dart) │ └── Chevron → record options sheet (Phone Mic record / Phone Call) ├── [settings gear, top right] → Settings sheet (settings_drawer.dart) — present on every slot │ ├── [slot 0] Home (home/home_content.dart) │ ├── Conversation capture widget, today's tasks widget │ ├── Daily Recaps → Daily Summary Detail; "View All" → Conversations │ ├── Mind Map → Memory Graph (memory_graph_page.dart) (only with ≥3 conversations) │ └── Get-started tiles (only with <3 conversations) │ ├── [slot 1] Conversations (conversations_page.dart) │ ├── Folder tabs (All, Starred, custom folders) │ ├── Top-bar extras on this slot: sync icon (when paired/pending), search, calendar date filter │ └── Conversation item (GestureDetector row) → Detail (conversation_detail/page.dart) │ ├── Transcript, Summary, Action Items tabs, share, audio (search hidden on Action Items tab) │ ├── Pull-down menu: Copy Transcript / Copy Summary / Copy ID / Share Audio (if audio) / Link Event / Test Prompt │ └── Back via in-app top-left IconButton — not the OS/adb back gesture │ ├── [slot 2] Tasks (action_items_page.dart) │ ├── Categories: Today, Tomorrow, Later, No Deadline, Overdue │ ├── FAB → Create task sheet (action_item_form_sheet.dart) │ ├── Task checkboxes, drag-drop reorder, task → goal linking │ └── Top-bar extras on this slot: export → Task Integrations, completed toggle │ └── [slot 3] Apps (apps/page.dart) — "Search 1500+ Apps" / "Featured" (explore_install_page.dart) ├── Popular apps (horizontal scroll) ├── Category sections → Category apps page ├── App item → App Detail (app_detail/app_detail.dart) │ └── Reviews, capabilities, install/enable, Chat button → Chat └── Top-bar "+" on this slot → Add App / Add MCP Server Settings sheet (settings_drawer.dart) — search + close header, then five visual groups: Account · Plan & Usage, Referral Program · Device … Data & Privacy · Help & About, Feedback · Developer Settings. Every top-level row has a ValueKey (`settings_account`, `settings_group_<group>`, or `settings_row_<SettingsDestination>` for the direct rows Plan, Referral and Feedback); each page has a Scaffold key (`settings_page_<page>`); rows on group/Account pages are `settings_row_<SettingsDestination>` (plus `settings_row_voiceResponseMode`, `settings_row_transcribeLater`, `settings_row_backgroundMode`, `settings_row_name`, `settings_row_email`, `settings_row_userId`, `settings_row_version`). ├── Account [settings_account] — shows the name, email as subtitle → Account page (profile.dart, ProfilePage) │ ├── Name → Change name dialog; Email (read-only) │ ├── User ID (tap copies) │ └── Sign Out → Confirmation dialog; Delete Account (delete_account.dart) ├── Plan & Usage [settings_row_planAndUsage] (usage_page.dart) — "Pro" value when paid ├── Referral Program [settings_row_referral] (referral_page.dart) — NEW tag ├── Device [settings_group_device] → settings_groups.dart │ ├── Device Settings (device_settings.dart) (only when a device is connected) │ ├── Offline Sync (sync_page.dart / auto_sync_page.dart) │ ├── Phone Calls (phone_call_settings_page.dart) │ └── Permissions (permissions_page.dart) — microphone, Bluetooth, notifications ├── Recording & Transcription [settings_group_recording] → settings_groups.dart │ ├── Transcription (transcription_settings_page.dart) — provider value; Language; Custom Vocabulary │ ├── Voice Profile → guided introduction (onboarding/speech_profile_widget.dart); Identifying Others (people.dart) │ ├── Voice Response (picker sheet); Conversation Timeout (picker) │ └── Recording (BETA): Transcribe Later switch; Background Mode switch (Android only) ├── Notifications & Display [settings_group_notifications] → settings_groups.dart │ └── Notifications (notifications_settings_page.dart); Home Screen; Conversation Display ├── Integrations [settings_group_integrations] (integrations_page.dart) — BETA; also opens from conversation detail ├── Data & Privacy [settings_group_privacy] → settings_groups.dart │ ├── Data Protection (data_privacy_page.dart); Memories (memories/page.dart) │ └── Export All Data (spinner while running); Import Data (import_history_page.dart) ├── Help & About [settings_group_help] → settings_groups.dart │ ├── Help Center → help.omi.me (Intercom platforms only) │ ├── What's New → Changelog sheet │ └── Version + copy button (iOS/Android) ├── Feedback / Report a bug [settings_row_feedback] → feedback.omi.me (Intercom platforms only) └── Developer Settings [settings_group_developer] (developer.dart) (Settings search finds every row above and opens the page that holds it.) Transcription Settings (transcription_settings_page.dart) — not in settings drawer; reached from Plan & Usage, Developer Settings, or the Plans sheet ├── Source toggle: Omi Cloud vs Custom STT ├── Provider selector, API key, model config └── Advanced JSON editors, logs viewer Phone Calls (phone_calls_page.dart) — from battery-widget phone icon, record options, or get-started tile ├── Phone Setup Intro when no verified numbers ├── Contacts / Keypad tabs; call → Active Call page └── Gear → Phone Call Settings (phone_call_settings_page.dart) — verified numbers list, delete button Plans sheet (plans_sheet.dart) — from Plan & Usage, chat quota-exceeded, phone-calls upsell Persona Profile (persona_profile.dart) — AI clone management ├── Avatar (100x100), name with verified badge ├── Share Public Link button ├── Make Public toggle └── 10 social link rows (omi, Twitter active; others Coming Soon) └── Twitter → Social Handle Entry → Verify Identity → Clone Success Connected Device (home/device.dart) — requires BLE ├── Device name, connection status, battery ├── Actions: Firmware Update, SD Card Sync, Disconnect, Unpair └── Device info: Product, Model, Manufacturer, Firmware, ID, Serial Voice Profile — guided introduction (onboarding/speech_profile_widget.dart, #14514) ├── Four sentence starters, phone mic, Next / Skip per prompt └── Review: edit or uncheck answers, then Save and finish (voice, memories, goal) ``` ### Widget Patterns **Bottom navigation bar:** - Android: 4 `InkWell` widgets at `bounds.y > 780`, sorted left-to-right by `bounds.x` - Detect with: `snapshot -i --json` → filter `flutterType == 'InkWell'` and `bounds.y > 780` - Navigate home: press the leftmost one - iOS (verified 2026-07-11, iPhone XR, 414pt-wide screen): 4 slots at y≈816, x=20/114/207/300, each w=94. Left to right: slot 0 = Home, slot 1 = Conversations (folder tabs All/Starred/…), slot 2 = Tasks, slot 3 = Apps marketplace ("Search 1500+ Apps" / "Featured") **Chat entry point (not a bottom-nav tab):** - Open chat by tapping the "Ask Omi" input bar on the home screen — a full-width gesture element directly above the bottom nav (~y=756, w≈382 on a 414pt-wide screen; verified iOS 2026-07-11) **Settings gear:** - Android: rightmost `button` widget in the top bar; detect by sorting buttons by `bounds.x` descending, take first - iOS (verified 2026-07-11): single top-right icon on home at ~x=362, y=58 → Settings sheet (Account, Plan & Usage, Referral Program, Device, Recording & Transcription, Notifications & Display, Integrations, Data & Privacy, Help & About, Feedback, Developer Settings) **Settings rows:** - `gesture` widgets with `bounds.width > 300` - Prefer the ValueKeys above (`settings_account`, `settings_group_*`, `settings_row_*`) over positions **Switch toggles:** - Type `switch` in snapshots - Press to toggle ON/OFF (no separate ON/OFF actions) **Bottom sheet pickers:** - Open when you press a settings row - Language items appear as `gesture` rows with `bounds.y > 380` - Many items — use scroll if needed **Conversation feed rows (iOS, verified 2026-07-11):** - `GestureDetector` widgets, h≈84–96; detail opens on tap - Go back with the in-app top-left `IconButton` (x=8, y=56, w=40) — not the OS/adb back gesture **Chat AI-message actions (iOS, verified 2026-07-11):**