smoketest

name

smoketest

description

Run a cat-factory Pi-agent smoketest end to end. Use when asked to smoketest, sanity-check, or shake out a model on real coding tasks through the actual Pi setup — e.g. "smoketest the new Cloudflare model", "does Llama get stuck on the healthcheck task", "shake out qwen-coder on the coding fixtures". Configures the matrix, runs cat-smoke against actual Cloudflare AI, and surfaces breakage / dead-ends / loops from the captured transcripts. It does NOT grade quality (that's the benchmark skill).

Smoketest runner

Drives a smoketest of the cat-factory Pi coding agent: configure → run the matrix → read the analysis → commit. A smoketest answers "can this model do real coding work through our Pi setup, and where does it get stuck?" — it captures the full prompt/response transcript and analyses it for breakage, dead-ends and non-productive loops. It does not rate quality; for that, use the benchmark skill instead.

1. Settle the config

From the user's request, decide the matrix:

• Models: the candidates to shake out (usually Cloudflare Workers AI models — the whole point is exercising actual Cloudflare AI through Pi locally).
• Fixtures: which coding tasks (default: all built-ins — add a /health endpoint + test, add a tested helper, scaffold a tiny service). They live in backend/internal/smoketest-harness/src/fixtures.ts. Point one at a specific repo by editing the fixture / the config's fixtures.

If a config exists (smoke.config.ts), use it. Otherwise copy backend/internal/smoketest-harness/smoke.config.example.ts to smoke.config.ts and edit models (and fixtures if narrowing). Confirm non-obvious choices with the user — runs cost tokens and the implementation flow is slow.

Prerequisites (check, don't assume)

• Cloudflare-AI candidates need CF_ACCOUNT_ID + CF_API_TOKEN; direct providers need their *_API_KEY. If a needed key is missing, say so instead of running cells that all error.
• The pi CLI must be on PATH. If it's absent, the run will report pi-not-runnable for every case — install Pi first.
• This is a local tool — never run it in CI.

2. Run the matrix

pnpm -C backend --filter @cat-factory/smoketest-harness smoke -- \
  run --config <abs-path-to>/smoke.config.ts --name <run-id>

Useful flags: --fixture <id> (one task), --relax-guard (let a looping run finish so it's captured whole instead of being killed at the guard threshold — use when investigating a loop). The command prints the run directory (docs/smoketests/<run-id>/) and a verdict tally; it exits non-zero if any case is broken.

3. Read the analysis

Open docs/smoketests/<run-id>/report.md — the verdict table + "what to look at first". For each non-healthy case, drill into cases/<case-id>/:

• analysis.md — verdict, findings (e.g. no-op-run, guard-no-edits, repeated-tool-call, terminal-model-error), metrics.
• transcript.md / transcript.jsonl — what the agent actually did (the full prompts + responses).
• diff.patch — what it changed.

Summarise for the user: which models are healthy, which are broken/degraded and why (cite the finding codes), and anything actionable (a model that never edits, a loop, an endpoint that 502s). For deeper qualitative read-through of a specific failing case, the investigate-telemetry skill is the production analogue.

4. Commit

Offer to commit the run directory as a record:

git add docs/smoketests/<run-id> && git commit -m "Smoketest: <run-id>"