| name | toastty-computer-use |
| description | Use this skill when a Toastty GUI bug, fix, or uncertainty needs human-like remote Computer Use interaction beyond the supported smoke tests, such as modal navigation, visual/focus checks, menu flows, onboarding, or reproducing ambiguous user reports with scripts/remote/computer-use-run.sh. |
Toastty Computer Use
Use this workflow for opportunistic GUI debugging and verification on the remote Mac when socket automation and smoke tests do not cover the interaction well.
This is not a CI-style assertion framework. A pass result means the Codex turn completed; only claim product verification when the transcript or final report contains the evidence you requested.
When to use this
- A GUI issue needs human-like navigation, judgment, or visual inspection.
- The flow involves modal sheets, onboarding, menus, focus, selection, drag-like interaction, or ambiguous reproduction steps.
- The existing smoke tests are too narrow or brittle for the behavior, and local foreground inspection would be disruptive.
Prefer scripts/remote/validate.sh --smoke-test ... when a supported smoke test covers the path. Use .agents/skills/toastty-dev-run/SKILL.md for ordinary live app launches, local Peekaboo checks, and deterministic remote smoke validation.
Before running
- Inspect the relevant diff or files and write down the exact behavior you need evidence for.
- Run
git status --short and choose the validation scope deliberately:
working-tree: current uncommitted local changes. Use this for validating a fix before commit.head: the current commit only. Prefer this for baseline checks.ref --ref <rev>: an explicit branch, tag, or commit.
- Pick a short run label:
<task-id-or-area>-<short-description>.
- Create a prompt under
artifacts/manual/computer-use/<run-label>.md.
Use sv exec -- for the runner so the remote GUI environment and any required service-managed configuration are injected. Do not probe TOASTTY_REMOTE_GUI_HOST outside sv exec.
Prompt templates
Start from one of these templates:
assets/ad-hoc-gui-check.md: repro, verify-fix, or exploratory GUI checks.assets/visual-inspection.md: visual, layout, focus, and text/label inspection.
Keep the prompt focused on 1-3 high-signal scenarios. The boundaries in the templates apply to the remote Computer Use turn, not to your local agent work.
Run
From the repo root:
mkdir -p artifacts/manual/computer-use
sv exec -- scripts/remote/computer-use-run.sh \
--scope working-tree \
--run-label "<task-id-or-area>-<short-description>" \
--timeout-seconds 600 \
--prompt-file artifacts/manual/computer-use/<task-id-or-area>-<short-description>.md
Use --scope head or --scope ref --ref <rev> when that better matches the question.
Typical timeout guidance:
300: one short inspection or simple repro.600: multi-step modal/menu verification.900: longer exploratory debugging. Prefer tightening the prompt before going higher.
Inspect results
Artifacts are copied back to artifacts/remote-gui/<run-label>/.
Decision tree:
- Read
result.json first.
- If
status is setup_error, inspect client-summary.json, remote/build.log, remote/app-server.log, and remote/app-server-session.log.
- If
status is timeout, treat artifacts as partial. Inspect remote/launch.json, remote/transcript.jsonl, and remote/app.log.
- If
status is agent_error, inspect client-summary.json.failureReason and remote/transcript.jsonl.
- If
status is pass, confirm client-summary.json.finalText or remote/transcript.jsonl contains the requested evidence. If the evidence is missing, report the run as inconclusive.
Useful files:
result.json: top-level status, model info, timing, and artifact index.client-summary.json: Codex app-server turn summary and failure reason.remote/transcript.jsonl: JSON-RPC transcript and Computer Use tool events.remote/app.log: Toastty runtime log for the isolated app instance.remote/build.log: remote build log.remote/launch.json: PID, runtime home, socket path, and app-server details.
Handoff
Report:
- Scope and run label.
- Result status.
- Whether the behavior was reproduced, verified, not reproduced, or inconclusive.
- Exact visible UI evidence and any uncertainty.
- Artifact path, for example
artifacts/remote-gui/<run-label>/result.json.
Do not turn an inconclusive Computer Use run into a claimed verification. If more confidence is needed, tighten the prompt, add an app-side assertion, or fall back to a deterministic smoke/test path.
