| name | investigate-scenario-flakiness |
| model | sonnet |
| description | Rank a run history's scenarios by cross-run flakiness and, on request, diagnose why each flaky one flips — one read-only report from `bajutsu flakiness` plus an optional `bajutsu triage --flaky` pass. |
Investigate scenario flakiness
Bind two existing commands into one procedure and return a single ranked report: which scenarios
actually flip their verdict at a constant fingerprint, and — when the caller asks for it — why.
bajutsu flakiness ranks; bajutsu triage --flaky explains. Running them separately means reading
the ranking, copying scenario names out of it, and retyping a triage invocation per name, which is
where a survey loses reproducibility: a different session picks a different cut-off and triages a
different set. This skill fixes the cut-off and the ordering so the same history yields the same
report.
It is read-only. It ranks, it reports, and it stops there — it never applies a fix, rewrites a
scenario, or re-runs anything. bajutsu triage's --apply / --write / --rerun are out of
bounds. Acting on the report is the caller's decision, whether that caller is a person or
investigate-ci-failure.
Nothing here reaches a pass/fail verdict. The classification is read from verdicts already recorded
by the deterministic runner (classify_stability in
bajutsu/analysis/audit.py); the optional triage pass is an
advisory diagnosis, never a judgment. Prime directive 1 holds.
Inputs
| Input | Meaning |
|---|
history | Directory of past runs, each holding a manifest.json. Omit to read the serve database instead |
org | Org whose runs to mine when history is omitted. Defaults to default |
limit | Cap on how many flaky scenarios the optional triage pass covers. Defaults to 5 |
use_ai | Run step 3's triage pass. Defaults to false — see the cost note below |
This skill does not assemble the history. It reads a directory (or database) that already
exists. A caller that must first collect runs — for example
investigate-ci-failure, which downloads them from CI
artifacts — does that assembly itself and hands over the finished directory.
Why use_ai defaults to false
bajutsu triage --flaky has no rule-based agent: cross-run diagnosis reasons over the delta
between passing and failing runs, so --ai is mandatory and the command exits 2 without it
(_flaky_triage in bajutsu/cli/commands/triage.py).
Step 3 therefore always spends ANTHROPIC_API_KEY credit. A caller opts into that cost explicitly;
it is never incurred by default. With use_ai false the skill still returns the full ranking, which
is what an unattended caller normally needs.
Steps
1. Rank the history
bajutsu flakiness --history <history> --json
With no history, read the database instead: bajutsu flakiness --org <org> --json.
The command exits 2 on a missing runs directory or an unconfigured database. That is a real failure
of the inputs, not an empty result — report it and stop rather than continuing with nothing.
Parse the JSON. Its shape is a FlakinessReport
(bajutsu/serve/flakiness.py): a scenarios array plus a
skipped count of runs that carried no fingerprint or no recorded verdict. Each scenario entry
carries name, scenario_hash, device_os, runs, passed, failed, flip_rate,
classification, representative_pass_run_id, and representative_fail_run_id.
name is representative, not a grouping key. rank_flakiness groups by
(scenario_hash, device_os) and scores the run-level verdict; name is only the first scenario
listed in that run's summary. The two coincide for a single-scenario run and diverge for a suite,
where the scenario that flipped may not be the one named — and a CI lane runs a suite. Check the
history before trusting name: when its runs hold more than one scenario, report the entry as a
run-level flip at that fingerprint rather than as a finding about the named scenario, and reach for
bajutsu audit --history — which keys on the scenario name and so groups per scenario — when a
per-scenario answer is what the caller needs.
2. Keep only the flaky ones
Filter scenarios to classification == "flaky" — the verdict flips while the content fingerprint
stays constant. Drop the other two classifications, and drop them for a reason worth stating in the
report rather than silently:
deterministic — every observed run agreed. A consistent failure is reproducible, not flaky, and
triaging it would return the obvious.unproven — fewer than two runs at this fingerprint. One run proves nothing either way.
The array already arrives flaky-first, then by descending flip_rate. Do not re-sort it. Take the
first limit entries as the triage set.
3. Diagnose each flaky scenario (only when use_ai is true)
Skip this entire step when use_ai is false, which is the default. Go to step 4.
When the caller passed use_ai, run one triage per scenario in the set:
bajutsu triage --flaky --ai --scenario <name> --history <history>
Three details about that invocation:
- Pass no
run_dir. --flaky ignores the positional argument entirely and reads the whole
history itself. Passing one is misleading, not harmful.
--history is required here too, even though step 1 already read it. The two commands take
the directory independently.- Never add
--apply, --write, or --rerun. They are what makes this skill read-only.
--scenario matches by substring, and triage resolves the exact name from the first run that
matches, so a name shared as a prefix by two scenarios never mixes them. Use the name exactly as
step 1 reported it.
On a multi-scenario history, qualify what this diagnosis covers. Per step 1's caveat, name is
the run's first scenario, while the ranking scored the run-level verdict. Triage therefore contrasts
the named scenario's own passing and failing runs, which on a suite may not be the scenario whose
failure flipped the run. Say so in the report rather than presenting the diagnosis as the cause of
the flip.
That gap also makes the "nothing to contrast" exit reachable here, so do not treat it as impossible:
the ranking can call a fingerprint flaky on run-level verdicts while the named scenario itself passed
in every run, leaving triage with no failing side. Report that as what it is — the named scenario is
not the one flipping — and point at the run-level entry, rather than reporting a clean result or a
changed history.
4. Handle an empty set
When step 2 leaves no scenario, return a report saying so, together with how many deterministic
and unproven scenarios were seen and the skipped count from step 1.
State this as a finding, not as silence. "No scenario is flaky in this history" and "flakiness was
never assessed" are different answers, and a caller acting on the report needs to tell them apart.
5. Report
Return one Markdown report. Per scenario, carry:
- the scenario
name and its classification
flip_rate, and the passed / failed / runs counts it derives fromrepresentative_pass_run_id and representative_fail_run_id, so a reader can open both sides'
evidencedevice_os when the entry carries one — a scenario ranked per OS is not the same finding as one
ranked across a fleet- the triage diagnosis, when step 3 ran
- whether the entry's runs held more than one scenario. If they did, label the entry a run-level
flip at that fingerprint and name
scenario_hash alongside name, so a reader is not told that a
particular scenario is flaky on evidence that only shows the run flipped.
Close with what was excluded and why: the deterministic / unproven counts, step 1's skipped
count, and how many flaky scenarios fell outside limit.
When step 3 did not run, say so explicitly and name the reason (use_ai was not set). A report
that simply omits the diagnosis column reads as "no cause was found", which is a different claim
from "no cause was looked for".
The report is the whole return value — there is no structured side-channel field. The same text
serves a person reading it directly and a calling skill parsing it.
