keel-flake-audit

name	keel-flake-audit
description	Detect intermittently-failing tests from recent CI history (or repeated local runs); dedupe against tracked flakes and open one tracking issue per newly-detected flake — routed to keel:ship.

keel-flake-audit

Use this skill when the user asks to run the keel command flake-audit (e.g. keel flake-audit ..., flake-audit <args>, or /keel:flake-audit). It reads every project value from .keel/project.yaml via the keel CLI.

/keel:flake-audit

Live progress — stamp this run (required)

So this run shows live on keel-visual's board, record it with keel activity as you go. This command's phases are: orient → evidence → aggregate → classify → dedupe → report → open. Pick one stable --run-id for the whole run (e.g. flake-audit-<issue-or-pr>):

• Right now, before the work below, stamp the first phase: keel activity .keel/project.yaml --root . --write --command flake-audit --run-id "$RUN" --phase orient
• Re-run with the next --phase (evidence, …) as you advance through the flow.
• At the end: keel activity .keel/project.yaml --root . --run-id "$RUN" --done

Treat this like any other contractual step — do not skip it. The one allowed exception is a core too old to ship keel activity (keel < 1.6.0): then skip it silently and never block the command.

Command step evidence

Every numbered step in this command is contractual. Complete the step, record the evidence it asks for, or explicitly mark it N/A — <reason> before moving on. If a step has an external side effect such as a GitHub comment, issue, review, report, branch, or PR, the side effect must be posted or written through the selected transport and cited in the final summary. Never silently skip a step because the runtime, agent, or prompt feels obvious.

Project-neutral flaky-test audit. Every project value — the CI workflows, the base branch, the test gate, the repo — is read from .keel/project.yaml via the keel CLI. The test command is the project's (keel run-gates .keel/project.yaml --root .); this adapter never names a specific test runner.

The command is read-only on test code: it never edits test sources, never auto-disables a test (no skip/ignore/only annotation), never reruns a test to "confirm" a flake, and never closes an existing flake issue. It aggregates per-test pass/fail signal, classifies tests whose failure rate crosses the threshold as flaky, dedupes against open flake issues, and opens one tracking issue per newly-detected flake. Re-runs MUST NOT spam duplicates — the Step 4 dedupe is load-bearing.

The defining rule: only an across-runs disagreement (a test that both passes and fails under identical conditions) marks a flake. A test that consistently fails is a real bug, not a flake — it belongs to /keel:ci-check and human triage; do not classify it here. All published artifacts (report, issue bodies) MUST be English.

Step 0 — orient + parse arguments + runtime gate

keel validate .keel/project.yaml --root .
keel plan     .keel/project.yaml --root .     # read base_branch, ci_workflows, repo
keel plan     .keel/project.yaml --root . --command flake-audit --live --json

The live plan is the operator-consent preflight. Before opening issues, routing fixes to /keel:ship, using secrets, publishing, or calling production-adjacent systems, parse contract.operator_consent; if requires_operator_consent is true, STOP and ask the operator to rerun with the required --approve-scope values.

Arguments:

• --days <N> — CI-history lookback window in calendar days (UTC). Default 14. Reject 0, negatives, non-integers.
• --runs <N> — when reading CI history is unavailable, re-run the project's test gate N times locally instead (default 5). The two evidence sources are mutually supportive: prefer CI history; fall back to repeated local runs.
• --threshold <P> — minimum failure rate to flag, a decimal in [0, 1]. Default 0.10. Reject anything outside [0, 1], non-numeric, or a missing value after the flag.
• --open-issues — open a deduped tracking issue per newly-detected flake (Step 6) and route each fix to /keel:ship. Without it, report-only.
• --dry-run — compute the report and print findings, but skip every issue-create mutation; print would create: <title> per skipped issue.
• Reject unknown flags.

Runtime-availability gate. Test-level CI-history analysis needs the host's check-run + artifact APIs (via the host CLI; no MCP equivalent today). If those are unreachable AND the project's test gate cannot be run locally either, exit cleanly with a single line saying flake detection requires either CI check-run/artifact access or a runnable local test gate, neither available in this runtime. Do not error or partial-run.

Step 1 — gather evidence

CI-history mode (preferred, full fidelity): enumerate the in-window CI runs of the project's ci_workflows on base_branch (drive it via commits in the window + their check-runs where a direct run-by-date listing is unavailable; document the substitution honestly in the report's Limitations). For each failing check run keep its conclusion, workflow name, start time, the run-page URL (for "sample run URLs"), and the test-report artifact URL if extractable.

Local-runs mode (fallback): run keel run-gates .keel/project.yaml --root . --runs times on a clean tree and record per-run pass/fail.

Step 2 — build per-test aggregates

• Test-level (full fidelity): parse each run's test-report into per-test results keyed by fully-qualified test name; track pass_count and fail_count. Capture, per failing test, the first ~5 lines of the failure signature from the most recent failing run.
• Degraded run-level (no per-test granularity — artifacts unreachable): aggregate only at the run level (which runs failed, on which commit, with which URL); skip per-test classification and per-flake issue creation, and produce a Limitations-flagged summary only.

Step 3 — classify flakes (test-level only)

A fully-qualified test with both passes and failures in the window is flaky when:

fail_count >= 3  AND  fail_count / (pass_count + fail_count) >= <threshold>

The fail_count >= 3 floor is load-bearing: it stops a single transient blip being labelled flaky. A test that never passed in the window is a deterministic failure, not a flake — do not classify it here.

Step 4 — dedupe against tracked flakes

Search open issues labelled flake (canonical title shape flaky test: <fully.qualified.name>) and build the tracked-name set by stripping that prefix. Carry forward only newly-classified flakes not already tracked. This dedupe is the only reason the command is safe to re-run on a schedule — do not weaken it.

Step 5 — build the report

Do not hand-write the tables. Assemble the structured flake data and let the renderer own the layout — the summary line, the newly-classified table (test · fail rate · failures · sample runs · signature, or _no new flakes above threshold_ when empty), the already-tracked (deduped) list, and the Limitations section — then post the rendered body verbatim:

keel render-report --kind flake-audit --payload flake.json > body.md

flake.json is one object:

{ "codename": "FLAKE-AUDIT-<DATE>-<UTC_TIMESTAMP>",
  "summary": { "runs": 0, "distinct": 0, "classified": 0, "opened": 0 },
  "new_flakes": [{ "test": "<fqn>", "fail_rate": "<f>/<t>", "failures": 3,
                   "samples": ["<url>", "..."], "signature": "<first lines, inline>" }],
  "tracked": [{ "test": "<fqn>", "issue": 123 }],
  "limitations": ["<honest caveat>"] }

Build new_flakes most-recent-sample-first, up to 3 sample URLs each. Omit tracked / limitations (or pass []) when empty; the renderer drops those sections. Codename pin (load-bearing): the codename field is the body's literal first line — FLAKE-AUDIT-<DATE>-<UTC_TIMESTAMP>, nothing above it — the FLAKE-AUDIT-<DATE>- anchor downstream consumers locate the run by. In degraded run-level mode (no per-test history), leave new_flakes empty and record the failing-run list plus the "test-level classification skipped" caveat under limitations.

Step 6 — open issues per new flake + route to ship

Only under --open-issues, and test-level mode only (degraded mode skips this). Per newly classified flake:

• --dry-run → print would create: flaky test: <name> and skip.
• Otherwise → open an issue titled flaky test: <fully.qualified.name> whose body carries the detecting codename + window, the fail rate (<fail>/<total>), the threshold, up to 3 sample run URLs, the fenced failure-signature excerpt, and a triage note: do NOT auto-disable; investigate root cause (timing, shared state, ordering, network); close with a fix PR or mark wontfix if the test is being retired.

Labels: always flake; add an area label tiered from tier3_globs / the test's path or FQN where the project's layout makes the area derivable; if the area is indeterminate, apply only flake and note that in the body. If a needed label does not yet exist, the create may fail to apply it — detect that, retry the create without the label, and add a Limitations bullet recommending the operator pre-create the label.

Hand each opened flake fix to /keel:ship (window + lock + review). Optionally suggest a quarantine annotation in the issue — but never silently skip a test from this command.

Step 7 — print

Always print the Step 5 report to stdout, even when not --dry-run — a silent run would force the operator to spelunk the tracker after the fact.

Stop conditions / invariants

• Never auto-disable a flaky test; never rerun a test to "confirm" flakiness (a rerun invalidates the very signal being measured) — only observed history counts.
• One flake = one issue — the Step 4 dedupe is load-bearing; re-runs never duplicate a tracked flake.
• Never close/edit/comment on existing flake issues — triage is a human call; this command only opens new ones.
• Per-step failure continues with what was fetched, documented under Limitations; only Step 0 arg-parse failure is fatal.
• No silent dry-run mutations — each create is printed as would create: <title> and skipped.
• Fail-soft · deterministic for identical inputs.