triage-flaky

name	triage-flaky
description	Use when a test is intermittently failing or you need to triage/quarantine flaky tests. Triggers on "flaky test", "triage flaky", "quarantine flaky test", "test fails intermittently". Reproduces with a 50x rerun harness, classifies flaky vs real regression, and quarantines with a ticket.

Triage Flaky Test — Autonomous Loop

Purpose: turn an intermittently-failing CI test into a confident verdict — quarantine-with-ticket or fix-PR — backed by a 50x reproduction harness, never re-triaging the same test twice.

Workflow

1. Read memory. Open loops/B-flaky-test-hunter/state/flaky-state.md. If the suspect is already in ## Quarantined, stop (do not re-triage). If it is in ## Watching, note the prior failure-rate trend — a rising rate means the test is getting worse. Checkpoint: you know whether this is new, watched, or already-quarantined.
2. Pull the suspect + CI history. Via the CI MCP connector, fetch the failing run, the test name, and the last N runs' pass/fail history and logs. Identify the exact test node id (e.g. path::test_name). Checkpoint: you have a concrete test id and its recent pass/fail pattern.
3. Spin a throwaway reproduction worktree. Create an isolated worktree off the base branch — NEVER on anyone's feature branch. This is disposable; it exists only to run the harness. Checkpoint: clean worktree at the suspect commit, nothing else touched.
4. Dispatch flaky-maker. In the repro worktree, the maker reruns the suspect test 50 times, computes the failure rate (failures / 50), classifies flaky-vs-real-regression, and drafts either the @flaky quarantine tag or a fix. Checkpoint: maker returns raw counts, the failure rate, a classification, and a draft action.
5. Dispatch flaky-checker (ADVERSARIAL, different model). The checker assumes the maker is wrong: it tries to prove the "flaky" label actually hides a real intermittent bug, independently re-checks the failure-rate stats, and returns an explicit CONFIRM/REJECT verdict with a confidence figure and evidence.