| name | run-evals |
| description | Prepare the environment and run the LLM-driven agent evals (e2e/agent-evals/) against a chosen sim-use binary. Use when the user runs `/run-evals` or asks to "run the agent evals", "run the LLM-driven tests", "eval the skill", or wants pre-release confidence that an agent reading the bundled skill still picks the right verbs. Costs real `claude -p` API calls — always confirm before spending. |
This skill orchestrates the agent-eval suite: natural-language cases executed
by a headless claude -p agent using the bundled skill (skills/sim-use/)
against the Playground fixture apps, judged by deterministic post-condition
checks. It verifies the layer the scripted E2E suites cannot: that an agent
reading SKILL.md reaches for the right verbs and survives the documented
pitfalls. A failure here with a green scripted layer usually means
skill-prose drift, not a CLI bug.
Execution is delegated to scripts/eval.sh / e2e/agent-evals/run.py — do
not reimplement their logic. Case anatomy, tags, and authoring rules live in
e2e/agent-evals/README.md. Run from the repo root.
Step 1: Decide WHICH sim-use is under test
The whole run — device probing, the agent's commands, the verification layer
— resolves sim-use from PATH unless overridden. Never let this be implicit:
- Ask (or infer from the user's request) which binary to evaluate:
- Installed release (default): whatever
sim-use resolves to on PATH.
- A development build: pass
-b <path>, e.g.
-b .build/out/Products/Debug/sim-use (SwiftBuild layout) or
-b .build/debug/sim-use (classic). Build it first with make build.
- Confirm the resolution and report it to the user before running:
python3 -c 'import pathlib,shutil; print(pathlib.Path(shutil.which("sim-use")).resolve())'
sim-use --version
The wrapper prints sim-use under test: <real path> (<version>) and the
run report records it under sim-use under test: — quote that line back
in your summary so the human knows exactly what was evaluated.
Step 2: Prepare devices and fixtures
For each platform you intend to cover (the wrapper auto-detects reachable
ones; use -p ios|android to restrict):
iOS
- Device Hub (Xcode 27) must be CLOSED —
pgrep dtuhidd must be empty. A
simulator booted while Device Hub is open has legacy HID disconnected;
sim-use's guard will (correctly) fail every case on it. If dtuhidd is
running: quit Device Hub, then shutdown && boot the simulator.
- Boot a simulator and wait:
xcrun simctl boot <UDID> && xcrun simctl bootstatus <UDID>.
- The Playground fixture must be installed. Check:
xcrun simctl listapps <UDID> | grep -c com.cameroncooke.SimUsePlayground
— if missing, install with scripts/test-runner.sh -b (builds sim-use +
Playground, ~2-3 min).
Android
- Start an emulator (not on PATH by default:
~/Library/Android/sdk/emulator/emulator -avd <AVD> &), wait for
adb shell getprop sys.boot_completed → 1.
- Both fixture packages must be present:
adb shell pm list packages | grep -c com.linecorp.simuse should be 2
(playground + device bridge). If missing, make e2e-android installs them.
- A stale bridge from an older CLI version is fine — the version parity
check fires and the agent is expected to recover via
sim-use android init (that recovery is itself part of what the evals exercise).
Step 3: Run
make eval
make eval ARGS="-y -t quick"
make eval ARGS="-p ios -b .build/out/Products/Debug/sim-use"
scripts/eval.sh -- --cases <id>
Cost: each case is a real claude -p agent (~1-3 min, real API charge; the
wrapper prints an estimate and asks unless -y). Never pass -y without the
user having approved the spend in this conversation.
Step 4: Interpret and report
Reports land in e2e/agent-evals/reports/<timestamp>/ (gitignored):
report.md (verdict table + env header), verdicts.jsonl, and one
stream-json transcript per case.
- All PASS → report the verdict table, the
sim-use under test line, and
the report path.
- FAIL → read the case's transcript before concluding anything. Classify:
- Skill-prose drift — the agent picked a wrong verb or missed a
documented pitfall the skill should have steered around → fix
skills/sim-use/SKILL.md, not the case.
- CLI regression — the right verb failed → treat as a product bug;
reproduce it directly with sim-use before filing.
- Environment/fixture noise — reboot-settling, Playground missing,
Device-Hub-poisoned boot → fix the environment and re-run; if the
coupling is inherent, tag the case
fragile (fragile-tagged cases never
gate a run).
- ERROR → the harness itself broke (reset failed,
claude missing);
fix the environment, don't touch cases.
Things to NOT do
- Don't run evals without stating which binary is under test.
- Don't pass
-y unless the user already approved the cost.
- Don't edit or delete eval cases to make a run green — a red case is signal;
classify it first (Step 4).
- Don't commit anything under
e2e/agent-evals/reports/ (gitignored on
purpose).
