android-a11y-agent

name	android-a11y-agent
description	Execute Android accessibility automation from natural language goals (open app, navigate UI, search, input text, and submit) using semantic UI understanding and stepwise planning with safety checks. Use when users ask to control Android apps by intent rather than fixed coordinates, especially for flows like entering search pages first and then typing.

Android A11y Agent

Convert natural language goals into reliable Android accessibility actions.

Preflight (MANDATORY before main execution)

1. Verify gateway whitelist includes required commands:
   • app.launch, screen.tap, text.input, ime.paste, ui.snapshot, ui.find, ui.click, ui.waitFor
2. Verify connected Android node declares the same commands.
3. If either check fails, stop main execution and return a fix plan (missing items + patch suggestion).

Output Contract (STRICT JSON)

Return only JSON:

{
  "intent": "...",
  "entities": {},
  "constraints": {},
  "riskLevel": "low|medium|high",
  "plan": [
    {
      "step": 1,
      "action": "ui.find",
      "args": {"query": "搜索"},
      "successCriteria": "找到可点击搜索入口或可编辑输入框",
      "fallback": "改用同义词: 查找/放大镜/搜"
    }
  ],
  "needUserConfirm": false,
  "confirmReason": "",
  "stopCondition": "达到目标页面即停",
  "explain": "一句话解释本次计划"
}

Planning Rules

1. Parse intent + slots from user goal.
2. Prefer two-stage search flow: find search entry -> enter search page -> locate editable input -> type/paste -> submit.
3. Match semantics in order: text > contentDescription > hint > viewId.
4. Add success criteria and fallback for every step.
5. Re-snapshot UI after key actions and re-plan if state is unchanged.
6. Require confirmation for risky actions (pay/send/delete/order-confirm).
7. Stop at confirmation page by default unless user explicitly asks to finalize.

References (load on demand)

• references/synonyms-zh.md: 同义词扩展（搜索、确认、返回等）
• references/risk-policy.md: 风险分级与确认策略
• references/retry-strategy.md: 重试预算、回退顺序、失败输出规范

Scripts

• scripts/plan_validator.py: 校验规划输出是否满足 STRICT JSON contract。
  • 用法：python scripts/plan_validator.py plan.json
  • 或：cat plan.json | python scripts/plan_validator.py
• scripts/preflight_check.py: 校验网关 allowCommands 与已连接节点 declared commands 是否满足运行前置条件。
  • 用法：python scripts/preflight_check.py --config config.json --node-status nodes-status.json
  • 返回：JSON 报告（missingAllowCommands / missingDeclaredCommands）

Execution Policy

• Use atomic actions only (launch/find/click/input/paste/back/wait).
• Retry with synonym/candidate switching up to 3 times.
• Abort with structured failure if budget exceeded.

Failure Contract (STRICT JSON)

{
  "ok": false,
  "failedAtStep": 3,
  "reason": "未找到可编辑输入框",
  "lastObservedState": "已进入搜索入口页，但输入框未聚焦",
  "nextSuggestion": "请用户手动点一下搜索框后重试"
}