mutation-testing

# Mutation Testing Role: worker. This command runs the mutation tool and triages survivors directly — it does not generate or estimate mutations, and it does not replace the tests it validates. Wraps a real mutation tool (Stryker, pitest, mutmut, Stryker.NET, go-mutesting) and adds AI triage of survivors. The tool generates mutations and reports survivors; the AI classifies survivors and writes fix tests. **Never estimate or guess mutation outcomes** — if no tool is available, help set one up; do not substitute reasoning for execution. This file describes the language-agnostic workflow and the data contract. **Per-language detail — install, run commands, timeout flag names, native-report mapping — lives in [`references/languages/`](references/languages/).** Detect the language first via [`references/tool-detection.md`](references/tool-detection.md), then load the matching language file. **Unattended, LLM-free overnight measurement.** [`scripts/mutation_nightwatch.py`](scripts/mutation_nightwatch.py) (detection/repair/measurement mechanics split into [`scripts/mutation_nightwatch_stacks.py`](scripts/mutation_nightwatch_stacks.py)) is a third, report-only mode alongside `mutation-kill`'s interactive and `--headless` fixing modes — no generation, no LLM, no commits, so it can run overnight unattended. See the companion [`mutation-night-watch`](../mutation-night-watch/SKILL.md) skill for launch/detach/schedule guidance. **Committed, human-approved exclude-policy bootstrap.** When no `mutation-exclude-policy.json` exists at the repo root yet, a night-watch run drafts one via [`scripts/mutation_exclude_policy.py`](scripts/mutation_exclude_policy.py) — the same two-signal judgment (score < 15%, NoCoverage > 50%, filename hint promotes to a stronger tier) `mutation-kill`'s own [Infrastructure exclusion detection](../../agents/mutation-kill.md#infrastructure-exclusion-detection-before-the-loop-starts) applies, but persisted as a source-controlled file instead of a build-output-only convergence log. The night-watch run only ever drafts a proposal alongside its other reports; `mutation_exclude_policy.py --approve` is the only code path that writes the committed file, and it is never called unattended. ## Constraints - **Always ask the user before running.** Present the time estimate and scope; get explicit approval. Mutation testing can be slow — never surprise the user. - Only run after tests exist; mutation testing validates tests, it does not replace them. - Do not chase 100% mutation score; equivalent mutants are noise. - Scope to changed files by default; full-codebase runs are periodic audits. - Surviving mutants in critical paths require action; in trivial code they may be acceptable. - **Per-mutant wall-clock timeout.** Every mutant run is capped at a wall-clock timeout. A run that exceeds the timeout counts as a **killed** mutant (matching the experiment-harness fix). Default: `timeout_seconds = max(60, suite_time_seconds × 10)`. Never run without a timeout — an infinite-loop mutant will hang the harness indefinitely. Per-tool flag names live in each [`references/languages/<lang>.md`](references/languages/). - **`--workflow-managed-approval` carve-out.** When this flag is set, the `Step 0` confirmation prompt is skipped — but the "always ask the user before running" invariant still holds at a higher boundary. The flag is reserved for orchestrated workflows that capture operator approval once for the whole run, then propagate the consent down to each scoped invocation. The authoritative caller registry is [`references/workflow-callers.md`](references/workflow-callers.md). Today's allowed callers are `/coverage-delta` (Phase 4 of `/test-improve`), `/quality-targets-converge` (Phase 6 of `/test-improve`), and `/test-improve` itself (Phase 2 baseline). Each inherits its workflow-level approval from `/test-improve` Phase 0. Any new caller must document where its workflow-level approval is captured before adopting the flag — see the registry file for the full process. ## Parse Arguments The skill accepts free-form natural-language arguments AND the following named flags for workflow callers: - `--scope <files-or-globs>` — restrict the mutation run to the listed files or shell globs. Comma-separated lists and quoted globs both accepted. When omitted, the skill scopes to changed files by default. - `--emit-json <path>` — write the structured result to `<path>` (see `## Machine-readable output` below) in addition to any human-readable output. The path's parent directory must be writable; failure writes to stderr and exits non-zero. - `--workflow-managed-approval` — skip the `Step 0` confirmation gate because the calling workflow captured approval at a higher boundary. Restricted to the allowlist in `## Constraints`. ## Time estimation See [`references/time-estimation.md`](references/time-estimation.md) for the formula, heuristic table, and how to estimate for a specific project. Present the estimate to the user; if > 5 minutes, suggest scoping down. ## Step 0: Confirmation gate Before any mutation run, present the estimated time and the scope, then block on stdin for explicit approval. The prompt is observable: stdout contains the literal string `Estimated time:` followed by the scope summary. This gate is **skipped when `--workflow-managed-approval` is set** — see the carve-out in `## Constraints`. ## Step 1: Detect or set up tooling **Prefer a local install** over a global one. Global installs depend on the user's `PATH` and produce silent "command not found" failures when it is not configured (see [`references/languages/csharp-stryker-net.md`](references/languages/csharp-stryker-net.md) for the observed Stryker.NET case). Each [`references/languages/<lang>.md`](references/languages/) file below shows the **local**-install command as the primary path. Use [`references/tool-detection.md`](references/tool-detection.md) to resolve the project's ecosystem to a mutation tool, then load the matching `references/languages/<lang>.md` for install and run commands. **Do not proceed without a working tool.** **Go is advisory-only.** When the project has a `go.mod`, resolve to **go-mutesting** in advisory mode (it is alpha quality — the surviving-mutant count is not a reliable gate). Advisory mode emits the `schema_version: 1` envelope with `"advisory": true`; orchestrated workflows treat that as **warn, do not block** — a non-zero survivor count never fails the gate. Always pair it with Go's built-in fuzzing (`go test -fuzz=FuzzXxx -fuzztime=30s ./path/to/pkg`), which is production-quality, for boundary and edge-case discovery. Full install path and fuzz idioms: [`references/languages/go-go-mutesting.md`](references/languages/go-go-mutesting.md). Never tell a Go project "no tool installed" without giving both the go-mutesting install path and the fuzz alternative. ## Step 1b: Configure per-mutant timeout Set a per-mutant wall-clock timeout before running. A timed-out mutant is **killed** (counts toward the mutation score as a non-survivor). This matches the experiment-harness fix in `docs/experiments/`. Derive the timeout: ``` suite_time_seconds = time the baseline test suite (from Step 1 output, or measure: `time <test-command>`) timeout_seconds = max(60, suite_time_seconds × 10) ``` For tool-specific flag names and config-file keys (e.g. Stryker's `timeoutMS`, pitest's `--timeoutConst`), see the matching [`references/languages/<lang>.md`](references/languages/). Document the chosen timeout in the output summary. ## Step 1c: Smoke gate — verify the tool is actually observing mutations Before running the full scan, run the tool against **one covered file** and confirm the mutation-switch mechanism is actually observing mutations at runtime. On some tool + test-framework combinations (issue [#554](https://github.com/bdfinst/agentic-dev-team/issues/554), [#557](https://github.com/bdfinst/agentic-dev-team/issues/557) on Stryker.NET + xunit.v3 + MTP), the tool cheerfully runs to completion but reports **every mutant as `Survived`** because the injected `ActiveMutation` env var never reaches the test host — the score looks like `0.00 %` but the run wasted hours. This gate catches that failure mode in one file's worth of wall-clock time. **Deterministic parse source.** Parse the tool's **report JSON** (Stryker's `StrykerOutput/<run>/reports/mutation-report.json`, pitest's `target/pit-reports/mutations.xml`, mutmut's `mutmut junitxml` output, etc.). **Do not parse stdout, the ANSI progress reporter output, or any log tail** — those are lossy, reporter-config-dependent, and don't survive redirection (see `## Capturing run output safely`). The gate is deterministic only when it reads the same artifact downstream steps read. **Three-way decision** (from the parsed counts): 1. **`Killed > 0`** — the mutation-switch mechanism is working. **Proceed** to Step 2's full run. 2. **`Killed == 0 && Survived > 0`** — the tool ran the test suite for every mutant and killed **none of them**. This is the mutation-switch-not-observing-mutations failure mode. **Halt** with an error message that: - names the failure mode (mutation-switch not observing mutations at runtime), - references issues #554 and #557 for the observed Stryker.NET / xunit.v3 / MTP case, - and enumerates the diagnostic checklist below. 3. **`Killed == 0 && Survived == 0`** (no-signal probe — all mutants `NoCoverage`, `CompileError`, or none generated) — the probe file provides no configuration signal. **Halt** and instruct the operator to pick a different probe file with real test coverage; a probe with no scored mutants can't distinguish "tool is broken" from "tool works but this file has no tests." **Diagnostic checklist** (for the mutation-switch failure mode — walk this before touching any config): - [ ] **Manual mutation kills the test?** Edit one covered line of the probe file, rebuild, run the specific test that covers it — does that test fail? If yes, the tests can observe changes but the tool's mutation-switch isn't activating them at runtime. - [ ] **`SolutionPath` in the config?** Multi-project configs that set both `SolutionPath` and an explicit test-projects list may see the tool enumerate additional test projects from the solution and prefer them over the ones listed. Verify the tool is actually running the test project you configured (`--diag` output on Stryker.NET; equivalent flag on other tools). See `references/languages/csharp-stryker-net.md` § SolutionPath trap. - [ ] **Unintended test-project enumeration?** Confirm no other test project in the solution is being picked up ahead of yours. If the tool is running the wrong test project, the smoke gate will fail even though the tool and configured tests are both fine in isolation. - [ ] **Is the target on .NET 10 with the default `vstest` runner?** Stryker.NET's bundled `vstest` runner can silently fail coverage capture on .NET 10 test assemblies, producing this exact `Killed==0 && Survived>0` shape. Retry with `-t mtp` — see `references/languages/csharp-stryker-net.md` § .NET 10 targets: default `vstest` runner can silently fake a 0% score. Per-language commands (which probe file to pick, how to invoke the tool with a single-file mutate glob, and where the report JSON lands) live in each [`references/languages/<lang>.md`](references/languages/). This step's decision procedure is language-agnostic; the mechanics are language-specific. **Mechanical enforcement (Claude Code sessions).** Inside a Claude Code session, the `PreToolUse` hook [`hooks/mutation_testing_smoke_gate.py`](../../hooks/mutation_testing_smoke_gate.py) enforces this step automatically — it intercepts `dotnet stryker` and shipped-wrapper invocations, checks for a smoke report at the fixed convention path `StrykerOutput/smoke/reports/mutation-report.json`, and blocks whole-scope runs until a probe report reports at least one `Killed` mutant. Run your smoke probe with `-O StrykerOutput/smoke` so the report lands where the hook looks; a single-file `--mutate` value (no glob metacharacters, no `;`) is recognized as the probe itself and never blocked. To bypass the gate for a legitimate one-off, set `MUTATION_SMOKE_GATE_SKIP=1` in the environment — every bypass appends one JSONL audit line to `.claude/metrics/gate-bypass.jsonl` recording the timestamp, cwd, and a sha256 hash of the command (the raw command is never logged). Direct-terminal and CI operators — outside Claude Code — rely on this step's prose plus the shipped wrapper's own defenses; the hook is Claude-Code-only. ## Step 2: Run the tool (scoped to target) Run scoped to user-specified files or changed files. Capture full output and note any HTML report paths. Per-language commands and scoping idioms — including the C# shard-aware execution path for large repos — live in [`references/languages/<lang>.md`](references/languages/). **Opt-in JS/TS generation-round savings.** For a JS/TS (Stryker) target, `mutation-kill`'s `--skip-static-mutants` flag excludes mutants that force a full-suite re-run from the survivor list handed to its generation step, trading a small, documented survivor over-count for fewer expensive-to-verify generation rounds (it does not change Stryker's own run time). Default OFF; scoring and convergence always read the unfiltered survivor count. See [`mutation-kill.md`'s Invocation section](../../agents/mutation-kill.md#invocation) and [`references/languages/javascript-stryker.md`](references/languages/javascript-stryker.md#static-mutant-skip-skip-static-mutants) for the mechanics. ### Capturing run output safely Do **not** wrap the mutation tool in a bare `<tool> 2>&1 | tee run.log` pipeline. Bash pipeline exit status defaults to the last command's — `tee` always exits 0 on a successful write — so any Stryker / mutmut / pitest / go-mutesting startup failure (missing tool manifest, invalid config key, wrong `DOTNET_ROOT`, compile-error abort) is silently masked. Downstream automation (background tasks, CI wrappers, this plugin's own monitor loops) then sees "success" and moves on, and the failure is discovered only when the report JSON is missing. Two safe patterns — pick by whether you need live tail: ```bash # One-shot run — direct redirect, simpler, no shell-option side effect. dotnet stryker --config-file stryker-config.json >StrykerOutput/full-run.log 2>&1 # Live-tail run — pipefail makes the pipeline exit the leftmost non-zero. set -o pipefail dotnet stryker --config-file stryker-config.json 2>&1 | tee StrykerOutput/full-run.log ``` This trap is portable across all languages the skill supports; the same rule applies to `npx stryker run ... | tee`, `mutmut run ... | tee`, `mvn pitest:mutationCoverage ... | tee`, and `go-mutesting ... | tee`. If you rely on `$?` or a monitor's exit-code trigger, always use one of the two safe patterns above. ### Probe file selection Before scoping the full run, pick a **probe file** — one file to shake out configuration and get a first honest signal. A good probe exercises both fast kills and the timeout ceiling, so the run tells you whether the tool is configured correctly. A bad probe produces a mass-CompileError smoke plume that validates nothing. Rules (language-agnostic): - **≥ 50 mutants** — enough operator variety that the score is not a coin flip. - **Highest existing mutation score in the target** — a file already well-tested by unit tests exercises both the "kill fast" and "timeout near the limit" paths; a weakly-tested file only measures how weakly it is tested. Avoid, in every language: - **Generated code** (Protobuf, OpenAPI stubs, ORM entity generators) — the mutation tool cannot distinguish generator output from hand-written code, and mutations on generated types typically fail to compile. - **DTOs / value objects** — no branching logic; every survivor is either equivalent or a wrapper-property assertion, so the file gives no signal about test quality. - **Files with near-0 % coverage** — validates configuration only, not test quality. Every mutant survives regardless of how the tests are written. Language-specific probe traps (particularly in C#/Stryker.NET, where certain operator combinations produce methods that don't exist) live in [`references/languages/<lang>.md`](references/languages/). ## Long-run inspection Real mutation runs take 15 min – several hours. During that window the tool emits progress on the terminal via an ANSI in-place reporter that **does not survive log redirection**, so a redirected run looks frozen even when the tool is fine. A silent-hang and a silent-config-error look identical to a healthy run until the summary lands. That's a several-hour feedback loop when the failure could have been caught in one. Every long run needs a periodic inspection loop watching **three signals**: 1. **Progress** — mutants tested / total. Read from a source that survives redirection: the tool's report JSON while in progress, a non-ANSI reporter's output (Stryker's `dots` reporter is the survives-redirection choice), or counts scraped from the log. Do NOT depend on the ANSI progress reporter. 2. **Health** — is the tool process still alive? Are child test-host processes alive? Elapsed wall-clock time. Silent + live = fine. Silent + dead = an unreported crash. 3. **Error inspection** — grep the log for known-broken signatures each tick, not just at the end. Failure modes to catch at tick boundary: - `Killed: 0` co-occurring with `Survived: > 0` — the mutation-switch-not-observing-mutations failure Step 1c gates against; if it slips past Step 1c (e.g. new file added mid-run), catch it here. - `CompileError` count spike — probe file, `mutate` glob, or generator-code inclusion misconfigured. - Tool-specific config-trap markers (Stryker.NET's `SolutionPath` naming a `.sln` outside the configured `test-projects` list, for example). - Frequency spikes in `Restarting` / `test process crashed` / `Timeout`. **Default cadence: 10 minutes** (600 s). Short enough to catch a stall or config error within one cycle, long enough not to spam operator output. Runs shorter than ~15 min don't need this — the summary at the end is sufficient. Cadence should be configurable. **This is a contract, not a mandated implementation.** Two examples the plugin ships: - **Portable Python wrapper** — a Python script that runs the tool as a subprocess, polls the log file, and emits one status + zero-or-more `[RED-FLAG]` lines per tick. Cross-platform via Python 3.10+ stdlib; works outside Claude Code (CI, direct terminal). The Stryker.NET reference in [`references/languages/csharp-stryker-net.md`](references/languages/csharp-stryker-net.md) documents the shipped wrapper (`csharp_stryker_net_wrapper.py`) + status loop (`csharp_stryker_net_status_loop.py`). - **In-session Monitor** — inside a Claude Code session, a `Monitor` tool call on the log file stream that emits an event on each recognized red-flag pattern. Cleaner integration but no coverage for out-of-session (CI, direct-terminal) operators. Whichever you pick, never implement the cadence as a foreground `sleep` loop. If the tick is driven by a scheduled wake-up rather than a stream monitor, see [`knowledge/long-run-waiting.md`](../../knowledge/long-run-waiting.md) for the calling contract — a wake-up armed without its replay instruction fails validation and schedules nothing, so the "inspection loop" stops inspecting after tick one. Per-language references may add tool-specific red-flag signatures — the language file lists them alongside the parse patterns. ## Step 3: Parse results Extract surviving mutants. Map each to: | Field | Source | | --- | --- | | File + line | Tool report | | Mutation operator | Tool report (`ConditionalBoundary`, `NegateConditional`, etc.) | | Original code | Read the source at that line | | Mutated code | Tool report or infer from operator | | Mutation score | Tool summary | ## Step 4: Triage survivors For each mutant, classify and act. **`NoCoverage` outranks `Survived`** — a survived mutant at least ran, so a tighter assertion can kill it; a no-coverage mutant was never reached at all, so writing a test that exercises the path is the higher-leverage move. | Classification | Meaning | Action | | --- | --- | --- | | **NoCoverage** | No test exercises this code path at all | Add a test that reaches the path before worrying about killing the mutant — coverage is the prerequisite | | **Equivalent** | Mutation produces identical behavior — no test could ever kill it | Mark `status: "equivalent"` with a `reason` string — excluded from the mutation denominator | | **Missing assertion** | Test executes the code but doesn't assert on affected output | Strengthen the assertion |