device-takeover

name

device-takeover

description

Take over and control external devices (Android phones, Linux handhelds, Linux PCs) via network. Provides screen vision, touch/input control, voice I/O, and app management. Use when: controlling a remote device, playing games on a handheld, navigating a phone, taking screenshots of external devices, sending touch inputs, installing apps remotely, voice-controlling a device. Triggers: 'take over', 'control the device', 'play on the device', 'navigate the phone', 'control my phone', 'device takeover', 'SuiPlay', 'android control', 'ADB', 'remote control device'.

Device Takeover

Control external devices on the local network: see the screen, send inputs, speak through speakers, hear through mic.

Supported Devices

Android (ADB)

• Setup: Enable Developer Options → USB/Wireless Debugging → pair via adb pair IP:PORT
• Screenshots: adb exec-out screencap -p > /tmp/screen.png
• Input: adb -s IP:PORT shell input tap X Y / input swipe / input text / input keyevent
• Apps: adb -s IP:PORT shell am start -n package/activity / pm install
• Shell: adb -s IP:PORT shell COMMAND
• See references/android.md for full command reference.

Linux Handheld (SuiPlay / Steam Deck)

• Setup: SSH access (password or key)
• Screenshots: Gamescope DBus: busctl --user call org.shadowblip.Gamescope /org/shadowblip/Gamescope/Wayland0 org.shadowblip.Gamescope.Wayland TakeScreenshot sy "/tmp/screenshot.png" 0
• Input: InputPlumber DBus: busctl --system call org.shadowblip.InputPlumber /org/shadowblip/InputPlumber/CompositeDevice0 org.shadowblip.Input.CompositeDevice SendButtonChord as 1 "BUTTON_EVENT"
• Voice out: ElevenLabs TTS → SCP mp3 → pw-play
• Voice in: pw-record → SCP → Whisper API
• See references/linux-handheld.md for button mappings and details.

Linux PC

• Setup: SSH access
• Screenshots: DISPLAY=:0 import -window root /tmp/screen.png or grim
• Input: xdotool / ydotool / evdev
• Voice: Same as handheld (PipeWire)

---

Android Setup — Complete Walkthrough

This is the most important section if you're connecting an Android phone. Follow these steps exactly.

Step 1 — Enable Developer Options

Developer Options is hidden by default on all Android phones. You need to unlock it first.

1. Open Settings
2. Scroll down and tap About Phone (on Samsung: About Phone → Software Information)
3. Find Build Number
4. Tap Build Number 7 times rapidly
5. You'll see a countdown: "You are now X steps away from being a developer"
6. After the 7th tap: "You are now a developer!"
7. It may ask for your PIN or fingerprint to confirm

Samsung note: On Galaxy devices, Build Number is under Settings → About Phone → Software Information → Build Number Pixel note: On Pixel devices, it's under Settings → About Phone → Build Number OnePlus note: Under Settings → About Device → Version → Build Number

Step 2 — Enable USB Debugging

1. Go back to Settings → Developer Options (now visible near the bottom of Settings)
2. Make sure Developer Options toggle is ON at the top
3. Find USB Debugging and toggle it ON
4. Confirm the dialog that appears

USB Debugging allows ADB to communicate with your phone over USB and WiFi.

Step 3 — Enable Wireless Debugging

This is what lets OpenClaw connect without a cable.

1. Still in Settings → Developer Options
2. Find Wireless Debugging and toggle it ON
3. Tap Wireless Debugging to enter its settings
4. You'll see: IP address and Port (this is your connection address)
5. Tap Pair device with pairing code — this gives you a one-time pairing code and a separate port number

Important: There are TWO port numbers:

• Pairing port — only used once during first-time setup
• Connection port — used every time you reconnect Write both down.

Step 4 — Install ADB on Your Mac/Linux Machine

# macOS (via Homebrew)
brew install android-platform-tools

# Ubuntu/Debian
sudo apt-get install adb

# Verify install
adb version

Step 5 — Pair Your Phone (First Time Only)

1. On your phone: go to Settings → Developer Options → Wireless Debugging → Pair device with pairing code
2. Note the IP:PORT and the 6-digit pairing code shown on screen
3. On your computer, run:

adb pair 192.168.1.XXX:PAIRING_PORT
# Enter the 6-digit code when prompted

You should see: Successfully paired to 192.168.1.XXX:PAIRING_PORT

Step 6 — Connect to Your Phone

After pairing, use the main connection port (not the pairing port):

adb connect 192.168.1.XXX:CONNECTION_PORT

You should see: connected to 192.168.1.XXX:CONNECTION_PORT

Verify it worked:

adb devices
# Should show your device listed as "device"

Step 7 — Test It

Take a screenshot and pull it to your Mac:

adb exec-out screencap -p > /tmp/screen.png
open /tmp/screen.png   # Opens in Preview on Mac

If you see your phone screen, you're connected. Done.

Reconnecting After Restart

The pairing is permanent. But you need to reconnect after every reboot (phone or computer):

adb connect 192.168.1.XXX:CONNECTION_PORT

The port number may change after a phone reboot. If connection fails, go back to Settings → Developer Options → Wireless Debugging to see the current port.

Troubleshooting

"adb: error: failed to connect" or "connection refused"

• Make sure your phone and computer are on the same WiFi network
• Check that Wireless Debugging is still ON (it can get disabled on reboot on some phones)
• Verify the port number — it may have changed

"unauthorized" in adb devices

• Check your phone screen — there should be a dialog asking "Allow USB debugging?" — tap Allow
• If no dialog appears, try: adb kill-server && adb start-server then reconnect

Developer Options disappeared

• Some phones hide it again after reboot. Go back to About Phone → Build Number and tap 7 times again.

"offline" in adb devices

• Run adb disconnect && adb connect IP:PORT to force a fresh connection

---

Gameplay Loop

For playing games on a remote device:

1. Screenshot the current frame
2. Analyze what's on screen (vision model)
3. Decide action based on game state
4. Send input (button press / tap / swipe)
5. Speak response if voice is active
6. Repeat

Keep screenshot→decide→input cycle under 3 seconds for responsive gameplay. Use scripts/game-loop.sh for the automated capture-analyze-act pipeline.

Voice Loop

For voice-controlled device interaction:

1. Record mic audio (5s chunks)
2. Transcribe via Whisper
3. Parse intent + generate response (GPT-4o, JSON mode)
4. Execute device actions from response
5. Speak response via TTS → device speakers
6. Repeat

Key Learnings

• Gamescope blocks X11 screenshots — must use its DBus TakeScreenshot method
• InputPlumber intercepts hardware gamepads — use SendButtonChord DBus, not raw uinput
• Gamescope only displays apps launched through PlayServe — flatpak from SSH won't render
• Android ADB is the cleanest path — full control with standard tooling
• Framebuffer capture works as fallback on Linux: sudo cat /dev/fb0 + ffmpeg rawvideo conversion
• Wireless Debugging port changes on phone reboot — always check current port in Developer Options