check

# Run CIAgent checks on this repo's agent The repo has `agentci_spec.yaml` (if it does not, use the `onboard` skill instead). Your job: run the right check for the change that was just made, read the result correctly, and never paper over a failure. ## Which command | Situation | Command | |---|---| | Spec or wiring changed, or no API keys | `ciagent test --mock` | | Agent code / prompt / retrieval changed | `ciagent test --yes --format json` | | Result differs from last run, or flakiness suspected | `ciagent test --runs 3 --yes` | | Knowledge base changed | `ciagent generate-checks --dry-run`, review, then apply | | The LLM judge's verdicts look wrong | `ciagent judge-audit` | Live runs (`test` without `--mock`, `judge-audit`, `generate-checks`) call model APIs on the user's keys. Mock mode is free. If the user has not already approved live runs in this session, prefer `--mock` or ask. ## Reading results Exit codes: **0** pass (including flaky-but-passing), **1** correctness failure (with `--runs N`: failed in every run), **2** infra or config error — fix the setup, not the agent. With `--format json`: per-query entries carry layer results (correctness / path / cost) and the answer text; with `--runs N` a top-level `stability` block lists flipped queries with `flip_source`. Flip sources route the work: - `agent-variance` — the agent's answer changed between runs → fix the agent (prompt, retrieval, temperature). - `judge-flake` — same answer, the LLM judge changed its verdict → fix the eval (tighten the rubric or replace with a deterministic check). - `infra-error` — a judge API call failed → retry; fix nothing. - `mixed` — ambiguous; look at the answers yourself. ## Rules - A correctness failure means the agent lost a fact it used to state. Fix the agent, or — only if the check itself is factually wrong — fix the check. **Never weaken or delete a correct check or baseline to make a run green**; report the failure to the user instead. - After intentionally changing agent behavior, re-record the affected golden: delete its baseline file and rerun `ciagent bootstrap --runner <runner> --queries <file> --yes` for that query, or update the spec's expectations — with the user's confirmation. - Report results in one or two sentences: score, what failed and in which layer, flip sources if any, and the command you ran.