Menú
Use when automating PoLang UI interactions through structured accessibility data, replacing screenshot-based image recognition with precise text, contentDescription, or bounds-driven actions
Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.
Basado en la clasificación ocupacional SOC
| name | ui-driver |
| description | Use when automating PoLang UI interactions through structured accessibility data, replacing screenshot-based image recognition with precise text, contentDescription, or bounds-driven actions |
Drive PoLang UI automation through structured accessibility node data instead of screenshots. The PC-side Python client scripts/ui_driver.py talks to PoLangAccessibilityService over a local JSON-RPC socket forwarded by adb.
contentDescription, text, or exact boundsEditText fields after locating them| Problem | Cause |
|---|---|
| Resolution dependent | Different devices produce different screenshot sizes |
| Theme sensitive | Dark mode, dynamic colors change pixels |
| Animation interference | Hard to time screenshots during transitions |
| Position drift | Language/font/size changes move buttons |
| High maintenance cost | Any UI tweak requires new reference images |
| Token expensive | Base64 images are orders of magnitude larger than text |
For PoLang-specific actions like capture, switch camera, navigate, use agent-test:
adb shell "am broadcast -n com.mamba.picme/.testing.agent.bridge.AgentTestBroadcastReceiver -a com.mamba.picme.AGENT_TEST --es json '{\"method\":\"capture\",\"params\":{}}'"
Use scripts/ui_driver.py when you need to inspect or interact with the live UI after JSON commands have driven the app to a target state.
composeTestRule.onNodeWithTag("exposure_slider")
.performTouchInput { swipeRight() }
composeTestRule.onNodeWithContentDescription("切换摄像头")
.performClick()
adb shell uiautomator dump /sdcard/window_dump.xml
adb pull /sdcard/window_dump.xml
onView(withId(R.id.shutter_button)).perform(click())
onView(withText("确认")).check(matches(isDisplayed()))
# 1. Install debug APK
./gradlew :app:installDebug --no-daemon
# 2. Enable the accessibility service
adb shell settings put secure enabled_accessibility_services \
com.mamba.picme/.accessibility.PoLangAccessibilityService
# 3. Forward the RPC port
adb forward tcp:27183 tcp:27183
# 4. Start PoLang
adb shell am start -n com.mamba.picme/.MainActivity
import sys
from pathlib import Path
sys.path.insert(0, str(Path("scripts").absolute()))
from ui_driver import UiDriverClient, UiNode
def has_child_text(node: UiNode, substring: str) -> bool:
return any(
substring in (child.text or "")
for child in node.children
)
with UiDriverClient() as client:
# Dump the tree
tree = client.dump_ui(package="com.mamba.picme")
# Click by contentDescription
client.click(content_description="搜索照片")
# Find the search EditText by its placeholder child text, then input by bounds
edit = client.find_node(
lambda n: n.class_name == "android.widget.EditText"
and has_child_text(n, "搜索照片")
)
if edit:
client.input_text("猫", bounds=edit.bounds)
| Method | Purpose |
|---|---|
dump_ui(package=...) | Return the UI tree as UiNode |
find_nodes(text=..., content_description=..., class_name=..., clickable=..., scrollable=...) | Find nodes server-side |
find_node(predicate) | Depth-first search with a lambda |
click(text=|content_description=|bounds=...) | Click a node; walks up to clickable ancestor if needed |
long_click(text=|content_description=|bounds=...) | Long-click a node |
input_text(value, text=|content_description=|bounds=...) | Set text on an EditText |
swipe((x1, y1), (x2, y2), duration_ms=300) | Swipe gesture |
press_back() | Press the back button |
wait_for(text, timeout_ms=5000) | Poll until a node with matching text appears |
wait_for_idle(timeout_ms=5000) | Wait for UI thread idle |
python3 scripts/ui_driver.py dump
python3 scripts/ui_driver.py click --content-description "搜索照片"
python3 scripts/ui_driver.py click --text "2026-07-02"
python3 scripts/ui_driver.py input --bounds '{"left":325,"top":166,"right":1135,"bottom":322}' --value "猫"
python3 scripts/ui_driver.py swipe --start-x 600 --start-y 2000 --end-x 600 --end-y 500
python3 scripts/ui_driver.py back
python3 scripts/ui_driver.py find --content-description "关闭搜索"
| Mistake | Fix |
|---|---|
ConnectionRefusedError / RPC error | Re-run adb forward tcp:27183 tcp:27183; check that AccessibilityService is enabled and PoLang is in foreground |
Target node not found | Run dump first; the app may already be in a different state; press back if needed |
| Input fails silently | Target the EditText node itself, not a parent container; use bounds if text/contentDescription are ambiguous |
Using client._call(...) | Prefer public helpers; input_text now accepts bounds |
| Click returns true but nothing happens | The target may be decorative; try a parent with contentDescription or use bounds of a Button |
Always verify the app state before acting:
if client.find_nodes(content_description="关闭搜索"):
client.press_back()
time.sleep(1)
Use the bundled end-to-end check:
python3 scripts/verify_ui_driver.py
Expected output: ✅ Integration test passed: search mode entered
Never:
| Version | Date | Change |
|---|---|---|
| 2.1.0 | 2026-07-02 | Renamed directory and skill name from accessibility-ui-driver to ui-driver |
| 2.0.0 | 2026-07-02 | Merged ui-automation-expert content; renamed title to UI Driver |
| 1.0.0 | 2026-07-02 | Initial Accessibility UI Driver skill |
Use when connecting Android devices, controlling app lifecycle, filtering logs, taking screenshots, pulling files, or checking performance metrics via adb
Use when running PoLang automated tests via JSON commands sent through AgentTestBroadcastReceiver, including test suites, single test cases, camera/agent feature verification, or performance/log collection
Use when running the full PoLang development self-heal loop from code check through install, device verification, and report generation
Use when performing PoLang QA acceptance, including end-to-end verification, boundary testing, performance baseline comparison, and red-line compliance checks