cat-factory

Run a cat-factory Pi-agent smoketest end to end. Use when asked to smoketest, sanity-check, or shake out a model on real coding tasks through the actual Pi setup — e.g. "smoketest the new Cloudflare model", "does Llama get stuck on the healthcheck task", "shake out qwen-coder on the coding fixtures". Configures the matrix, runs cat-smoke against actual Cloudflare AI, and surfaces breakage / dead-ends / loops from the captured transcripts. It does NOT grade quality (that's the benchmark skill).

Investigate a failed or suspicious agent run (implementer/coder, bootstrap, reviewer, etc.) using the production telemetry in D1. Use when asked to look into a run failure, "no progress" abort, a stuck/looping agent, model output quality, token/truncation issues, or "what went wrong with the latest run". Pulls the run lifecycle from `agent_runs` and the per-LLM-call prompt/response/usage from `llm_call_metrics`, then reads the tool-call loop to find the root cause.

Run a cat-factory agent benchmark end to end. Use when asked to benchmark, compare, or measure models or prompt versions on the agent tasks (requirement review, code review, implementation) — e.g. "benchmark Llama vs Claude on code review", "compare review@v1 vs a new prompt", "run the benchmark and grade it". Configures the matrix, runs cat-bench, invokes the benchmark-arbiter skill to grade, merges the report, and offers to commit under docs/benchmarks/.

Grade cat-factory benchmark runs. Use when asked to grade, score, judge, or arbitrate a benchmark run produced by the benchmark-harness (cat-bench) — typically a directory under docs/benchmarks/ containing a grading/ folder. Reads each cell's candidate output against its rubric and writes grades.json, grading-summary.md, and conclusions.md.