flaker-setup

# flaker setup skill `@mizchi/flaker` (0.14.0) is a test-intelligence CLI with a declarative apply model: `flaker.toml` describes the desired state, and `flaker apply` reconciles the repo to that state by running its `collect_ci` / `calibrate` / `cold_start_run` / `quarantine_apply` actions in the right order based on current DB state and repo probe. Callers do not memorize the sequence. ## Existing flaker setup? Upgrade first If the repository already has `flaker.toml`, or its CI already calls `flaker`, this is an upgrade, not a setup. Signs of a pre-0.13.0 setup: `[profile.*]` sections, `run --profile`, `FLAKER_PROFILE`, `apply --target` / `--emit`, `flaker ops|collect|analyze|policy|gate …`, `adaptive = true`, `strategy = "random"|"gbdt"|"coverage-guided"`. A 0.13.x setup needs no config rewrite for 0.14.0, but scripts that pass `flaker query` a write, several statements, or a file (`FROM 'x.csv'`) now fail; the agent changelog lists them. Read `${CLAUDE_PLUGIN_ROOT}/docs/agent-changelog.md` (GitHub: <https://github.com/mizchi/flaker/blob/main/docs/agent-changelog.md>) and follow it: it has a one-shot grep for every old form, the exact error lines with fixes, a rewrite map resolved to the current command, and a verify checklist. Fix config, env vars, workflows and `package.json` scripts together; a config that loads can still leave CI calling removed commands. **Always read the canonical checklist first.** It lives next to this skill in the plugin: - Plugin-relative: `${CLAUDE_PLUGIN_ROOT}/docs/new-project-checklist.ja.md` or `${CLAUDE_PLUGIN_ROOT}/docs/new-project-checklist.md` - GitHub: <https://github.com/mizchi/flaker/blob/main/docs/new-project-checklist.ja.md> or <https://github.com/mizchi/flaker/blob/main/docs/new-project-checklist.md> If both are unreachable, fall back to the procedure below. ## When this skill applies - "新しいプロジェクトに flaker を入れたい" - "flaker.toml を作って" - "GitHub Actions に flaker を組み込んで" - "this project should use flaker" - "flaker のセットアップ手順を教えて" ## Mental model: desired state + reconciler 1. User writes `flaker.toml` (`[gate.iteration|merge|release]`, `[promotion]` thresholds, `[quarantine].auto`). 2. `flaker plan` shows what `apply` would do right now. 3. `flaker apply` executes the plan (idempotent; safe to re-run). 4. `flaker status` shows drift vs `[promotion]` thresholds. The Day 1 flow is `flaker init → flaker doctor → flaker apply → flaker status`. The old imperative chain (`init → collect → calibrate → run`) no longer exists: `collect` was removed, one-off collection is `flaker import --ci`, and one-off calibration is `flaker calibrate`. ### Minimal declarative `flaker.toml` ```toml [repo] owner = "your-org" name = "your-repo" [adapter] type = "playwright" [runner] type = "playwright" command = "pnpm exec playwright test" flaky_tag_pattern = "@flaky" [affected] resolver = "workspace" [quarantine] auto = false # Day 1 recommended: keep false until history accumulates; # flip to true in Week 1-2 once `flaker status` drift shows moderate+ confidence. flaky_rate_threshold_percentage = 30 min_runs = 10 [gate.iteration] strategy = "affected" max_duration_seconds = 60 fallback_strategy = "weighted" skip_flaky_tagged = true [gate.merge] strategy = "hybrid" sample_percentage = 30 skip_flaky_tagged = true [gate.release] strategy = "full" # [promotion] is OPTIONAL — defaults (matched_commits_min=20, FNR<=5%, correlation>=95%, # holdout_fnr<=10%, data_confidence_min="moderate") apply when omitted. # Override only with explicit justification. Example: # [promotion] # matched_commits_min = 30 # data_confidence_min = "high" ``` `flaker init` generates a starter toml including `[gate.*]` defaults; expect to edit `[affected].resolver` before the first `flaker apply`. Recommend `flaker calibrate` once enough history has accumulated to write a tuned `[sampling]` block back to `flaker.toml` — it replaces the old manual `adaptive = true` toggle, which no longer exists. ## Optional: connect a test selector (jev-test-filter) Only when the user asks for it. flaker never selects tests for jev; it stores jev's decisions, checks them against full runs on the same commits, and hands jev a context (gate values, quarantined tests, per-test hints). Follow `${CLAUDE_PLUGIN_ROOT}/docs/jev-test-filter-integration.md` (or `.ja.md`); the short form: ```toml [selector] type = "jev" # recall_target = 0.90, min_failures = 20, max_hinted_tests = 200 by default [workflow_lanes] "nightly.yml" = { lane = "full-batch", full = true } # calibration needs full runs on the commits jev scored ``` ```bash flaker export --projection jev-context -o .flaker/context.json jev-test-filter --base main --context .flaker/context.json --exec -- vitest run # needs TYPESAFE_API_KEY flaker import .jev-test-filter --adapter jev flaker import --ci flaker calibrate --selector ``` Calibration only learns from commits that have both a jev record and a full run, so a PR-only jev job plus a nightly full run never overlaps. Run jev on pushes to main with a full run on the same commit, or run the full suite after jev on some PRs (the guide's "Put jev and a full run on the same commits"). In CI, check out with `fetch-depth: 0` and pass `--base origin/main`; for `vitest`, set `[adapter].artifact_name` to the artifact the full run uploads (`vitest run --reporter=json --outputFile=report.json`). Upload only the full-suite report under the full lane; the tests jev selected are a partial run. `flaker apply` does not run the selector steps, so run `import --adapter jev` and `calibrate --selector` after it. For a ready workflow (cache-backed database, `concurrency` group, jev on main with a full run on the same commit), start from `${CLAUDE_PLUGIN_ROOT}/examples/github-actions/jev-loop.yml`. Add `.jev-test-filter/` to `.gitignore`. Never put `cutoff`, `unsure_below` or `unsure_margin` in `flaker.toml`: the loader rejects them, and the current gate is the latest row of `flaker_v1.gate_calibration`. ## Decision points to confirm before touching files Ask the user (or infer from `package.json` / `pnpm-workspace.yaml` / repo layout) — do NOT guess silently: 1. **Adapter** — `playwright | vitest | jest | junit`. Look at `package.json` `devDependencies` and existing test files. Default to vitest for TS libraries, playwright for e2e, junit for non-Node. 2. **Runner** — `vitest | playwright | jest | actrun`. Usually matches the adapter. `actrun` wraps a GitHub Actions workflow file when local execution should mirror CI exactly. 3. **Resolver** — `workspace | glob | bitflow | git`. Pick `workspace` if `pnpm-workspace.yaml` or `package.json` `workspaces` exists. Pick `glob` for legacy single-package repos (then create `flaker.affected.toml`). Pick `bitflow` only if the repo already uses bitflow. 4. **CI history availability** — does the repo already have GitHub Actions runs? Either way, `flaker apply` handles it: with no history it skips calibrate and runs a cold-start gate; with history it pulls + calibrates + applies quarantine. 5. **GITHUB_TOKEN scope** — `gh auth status` must show `actions:read`. If missing, the user runs `gh auth refresh -s actions:read` themselves. ## Phase order ``` Day 0 prerequisites 5 min node>=24, pnpm>=10, gh auth, git remote Day 1 install + init 10 min pnpm add -D @mizchi/flaker → init → doctor Day 1 first apply 5 min flaker plan → flaker apply → flaker status Day 3 package.json scripts 5 min flaker:plan, flaker:apply, flaker:status, flaker:run:* Day 5 Actions integration 15 min cron `flaker apply` + PR advisory `flaker run --gate merge` Week 1 daily observation - flaker status (drift → promotion-ready signal) Week 2-4 promote to required - drift == ready, remove continue-on-error ``` Day 2 and Day 4 are intentionally empty — apply is idempotent, so there is no forced action between the Day 1 bootstrap and the Day 3 / Day 5 integration steps. Run `flaker apply` whenever you want (manually or in cron); skip days if nothing changed. **Never skip the `continue-on-error: true` on the first PR job.** The CI job becomes a required check ONLY after `flaker status` drift reports `ready` (all 5 `[promotion]` thresholds met). Promoting too early causes false negatives that erode developer trust. ## Day 1 commands (copy-paste ready) ```bash # 0. prerequisites node --version && pnpm --version && git remote -v && gh auth status # 1. install pnpm add -D @mizchi/flaker # 2. init (init now writes [gate.*] defaults) pnpm flaker init --adapter <adapter> --runner <runner> # 3. doctor pnpm flaker doctor # 4. edit [affected].resolver in flaker.toml (workspace | glob | bitflow) # 5. preview + converge export GITHUB_TOKEN=$(gh auth token) # optional if no CI history yet pnpm flaker plan pnpm flaker apply pnpm flaker status ``` ## What `flaker apply` does The planner produces a `PlannedAction[]` based on current state: | Action | Condition | |---|---| | `collect_ci --days 30` | `GITHUB_TOKEN` present. Pulls new CI runs. | | `calibrate` | `data.confidence` is `moderate` or `high`. Tunes `[sampling]`. | | `cold_start_run` (iteration gate) | No local history yet. Seeds the first local sample. | | `quarantine_apply` | `[quarantine].auto = true` AND history is usable. Applies the current quarantine plan. | When actions don't apply (e.g. no `GITHUB_TOKEN` on a brand-new repo), apply silently skips them. It is safe to run anywhere in the lifecycle. ## package.json scripts to add ```jsonc { "scripts": { "flaker": "flaker", "flaker:plan": "flaker plan", "flaker:apply": "flaker apply", "flaker:status": "flaker status", "flaker:run:iteration": "flaker run --gate iteration", "flaker:run:release": "flaker run --gate release", "flaker:eval": "flaker status --markdown", "flaker:doctor": "flaker doctor" } } ``` ## GitHub Actions snippets PR advisory (MUST be `continue-on-error: true` for the first 2-4 weeks): ```yaml - name: Run tests via flaker (advisory) run: pnpm flaker run --gate merge continue-on-error: true env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` Nightly apply (single scheduled cron replaces the old collect + run pair): ```yaml - run: pnpm flaker apply env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: pnpm flaker status --markdown > .artifacts/flaker-status.md ``` ## Promotion criteria for required check Only promote `flaker run --gate merge` from advisory to required when `flaker status` drift shows `ready` (all 5 `[promotion]` thresholds met), or equivalently when `flaker status --gate merge --detail --json` reports: - `Matched commits ≥ 20` - `False negative rate ≤ 5%` - `Pass correlation ≥ 95%` - `Holdout FNR ≤ 10%` - `Data confidence: moderate` or `high` The thresholds live in `[promotion]` of `flaker.toml`; defaults match the above. Override in config if the project needs stricter or looser gating. The simplest readout is `flaker status` — the drift section shows `ready` or lists unmet thresholds. If the user wants to gate sooner, push back: empirically less than 20 matched commits gives unstable readings. ## Pitfalls (encountered in real dogfood) | Symptom | Cause | Fix | |---|---|---| | `flaker.toml uses removed or renamed keys` (exit 2) | Config from 0.12.x or earlier (`[profile.*]`, `adaptive*`, removed strategies) | Follow `docs/agent-changelog.md` → "Errors you will see"; each error line names the key | | `flaker.toml uses deprecated keys` | Config from 0.1.x or earlier | Apply rename table from `docs/how-to-use.md#config-migration` | | `unknown option '--profile'` / `unknown command 'ops'` in CI | Workflow written for 0.12.x or earlier | Rewrite with the Commands table in `docs/agent-changelog.md` | | `Config file not found` | Wrong cwd | `cd` to repo root containing `flaker.toml` | | `flaker.toml sets values that belong in the database` | `cutoff` / `unsure_below` / `unsure_margin` under `[selector]` | Delete them; run `flaker calibrate --selector` instead | | `calibrate --selector` keeps: `no selector record has a full run on its head commit yet` | No full run on a commit jev scored (jev only on PR heads, full runs only on main), or a hand-imported report without `--commit` | Put jev and a full run on the same commits (see above); mark the full lane `full = true`; pass `--commit <sha>` to `flaker import <report>` | | `calibrate --selector` keeps for weeks; rationale starts `only N real failures observed; loosening needs…` or `recall lower bound … is below the target` | Expected: loosening needs 35 caught real failures at `recall_target = 0.90` | Leave `recall_target` alone; check `decision.real_failures` and `without_full_run` in `flaker calibrate --selector --dry-run --json` | | `flaker apply` aborts with `GITHUB_TOKEN` missing | Planner included `collect_ci` but env var absent | `export GITHUB_TOKEN=$(gh auth token)` and re-run | | `actrun runner requires [runner.actrun] workflow` | Missing actrun config | Add `[runner.actrun] workflow = ".github/workflows/<file>.yml"` | | `hybrid` selects 0 tests | Resolver not configured | Set `[affected].resolver` | | `flaker apply` stops after `collect_ci` | Calibrate or cold-start failed | Inspect the `ok/fail` lines; fix and re-run apply (idempotent) | | `flaker status` drift: `data_confidence` unmet | < ~10 matched commits | Wait, or run more `flaker apply` after CI accumulates | | Tests timeout in parallel | DuckDB single-writer | Serialize commands sharing the same database file (`[storage].path`, `.flaker/data` by default) | | `dist/moonbit/flaker.js` missing | Custom build environment | Should not happen with npm install — investigate package.json `files:` | ## Anti-patterns - **Do not** edit config keys to old names ("looks cleaner") — the loader hard-fails on legacy keys. - **Do not** run `flaker calibrate` until at least 30 commits of history exist. The recommended `[sampling]` needs FNR data to converge; there is no `adaptive = true` toggle to fall back on — that key was removed in 0.13.0. - **Do not** set `holdout_ratio > 0.2` — wastes runner time. - **Do not** skip `flaker apply` / `flaker calibrate` and hand-tune `[sampling]` — the calibrated values outperform manual settings in 90% of cases. - **Do not** make the PR job required before `flaker status` drift reports `ready`.