run-evals

# Skill: Run Evals Launch a real iPolloWork app and run coded eval flows against it. This skill owns **launch + run**; the prove/repair/verdict loop and evidence standard live in the **`fraimz` skill** — load that too for anything that ends in a verdict. ## Prerequisites - `daytona` CLI installed and logged in (`daytona login`), right org selected (`daytona organization use "<org-name>"`) - `.devcontainer/` files present in the repo - Optional provider coverage: reusable secrets volume populated once with `bash .devcontainer/setup-daytona-secrets-volume.sh .newtoken` (never print keys; sandboxes source every `/daytona-secrets/*.env` before Electron starts) ## Preferred path: Daytona sandbox ```bash daytona organization use "<org-name>" bash .devcontainer/test-on-daytona.sh <branch-or-commit> --artifacts-volume ``` The helper creates a fresh VNC-capable sandbox from the `ipollowork-eval-vnc` snapshot, mounts the secrets + pnpm-store volumes, starts XFCE/noVNC, Vite, and Electron with Daytona-safe flags, waits for CDP, then prints the CDP and noVNC URLs. `--artifacts-volume` mounts `/daytona-artifacts` served on port 8090 for published frame proof. Refresh the snapshot when dependencies change: `bash .devcontainer/create-daytona-ipollowork-snapshot.sh`. Verify the endpoint before running flows: ```bash curl -fsS "<CDP_URL>/json/list" # must include an iPolloWork page target ``` If it fails, inspect `/tmp/electron.log` — the real success marker is Chromium's `DevTools listening on ws://127.0.0.1:9825/...`. If the app shows the Welcome page, use a coded onboarding flow or create `/workspace/hello` through the visible UI before running a workspace-dependent flow. ## Run the flows ```bash pnpm evals --list pnpm evals --flow <flow-id> --cdp-url <printed-electron-cdp-url> pnpm evals --all --stack den # brings up MySQL + den-api + seed for cloud flows ``` The runner produces machine-checkable assertions, validated screenshots, and writes `fraimz.html` + `report.md` / `report.json` under `evals/results/<run-id>/`. If no coded flow exists for the behavior, add one in `evals/flows/<id>.flow.mjs` (see the `fraimz` skill and `evals/README.md` for the `ctx.*` API); use manual browser tools only to debug or prototype — a coded flow is the PR evidence. ## Recording (motion only) Frame proof is the default deliverable; record video only when motion matters (streaming, animations). Start with `bash .devcontainer/test-on-daytona.sh <branch> --record-video --recording-name <name>`, stop with `daytona exec "$SANDBOX" -- 'bash .devcontainer/stop-daytona-recording.sh'`, download via the port-8090 artifacts URL. Details: `daytona-recording-artifacts`. ## Local fallback When Daytona is down or quota-limited: ```bash pnpm install pnpm --filter @ipollowork/app typecheck IPOLLOWORK_ELECTRON_REMOTE_DEBUG_PORT=9826 pnpm dev # then: pnpm evals --flow <flow-id> --cdp-url http://127.0.0.1:9826 ``` Report clearly whether the result came from Daytona or the local fallback — a local run is not a Daytona validation. ## Teardown ```bash daytona delete "$SANDBOX" ```