harness-audit

# Harness Audit Role: orchestrator. This command analyzes harness effectiveness — it does not modify agents or configuration. You have been invoked with the `/harness-audit` command. > **Not `/agent-readiness`.** This audits the **dev-team plugin's own harness** > (review-agent effectiveness, model tiers, orchestration) from accumulated > runtime metrics in `metrics/`. `/agent-readiness` scores the **subject > repository's** readiness for AI-assisted development from a static checkout. > Different subject, different input, different output. The inward-facing > companion here is `/session-review`, whose `session-digest.jsonl` this command > consumes (Step 1). **Portable executable, repo-relative data (#1653).** `eval_ablation.py` now ships under `plugins/dev-team/scripts/` — its `--find-latest`/`--jsonl` mode is a generic JSONL reader with no repo-shape assumption, unlike the module's `--mode knowledge`/`--mode agent` paths, which stay monorepo-only (they read `evals/` fixtures never shipped). The call below is therefore `${CLAUDE_PLUGIN_ROOT}`-qualified like any other shipped script; only the `--jsonl` argument stays bare — `metrics/eval-ablation.jsonl` is genuinely *this repo's own* runtime metrics stream, consistent with the note above that this command audits the dev-team plugin's own harness. ## Orchestrator constraints 1. **Do not modify agents or configuration.** Produce a report only. All remediation requires human action. 2. **Write the report to a file.** Present only the summary table and next-steps in chat — do not repeat the full report. 3. **Be concise.** Use tables and short sentences. No preambles, no filler. ## Parse Arguments Arguments: $ARGUMENTS - `--output <path>`: Write report to a specific path. Default: `.dev-team-reports/harness-audit-<date>.md` - `--pdf`: After writing the report, render it to a sibling PDF via `hooks/lib/report_pdf.py`, resolving against the **actual** report path written this run (the `--output` override when given, else the default). See `knowledge/report-pdf-integration.md`. Additive; non-fatal if no engine is available. ## Steps ### 1. Check for metrics data Read metrics JSONL files from `metrics/`. Full field reference for every stream below: `${CLAUDE_PLUGIN_ROOT}/knowledge/telemetry-schema.md` — read it instead of re-deriving a schema from the emitter. Five complementary streams exist: - `metrics/*-task-log.jsonl` — **self-reported** task logs (whatever the model chose to record about itself). - `metrics/session-digest.jsonl` — **ground-truth** real-session digests from `/session-review` (#129): token/cost trends, `rework`/`accuracy` counts, and `utilization.never_observed_*`. Prefer this where it disagrees with the self-reports, and use `never_observed_*` to corroborate stale-component flags. Schema + join: see `docs/eval-system.md` → "Session-review trend digest". - `~/.claude/metrics/artifact-usage.json` — **per-artifact usage index** written by the telemetry hook on each Skill invocation. Use `last_used_at` to identify artifacts that have never been observed (absent from the index) or are stale (absent or `last_used_at` > 30 days ago). Cross-reference with `never_observed_*` in `session-digest.jsonl` for corroboration. See `knowledge/artifact-lifecycle.md` for the lifecycle threshold definitions. - `metrics/boundary-events.jsonl` — **boundary-level (policy-gateway) events** (#859): every guard hook's `block`/`warn`/`bypass` decision plus `intervention` keywords, each with the emitting `hook` and a `matched_rule` rule ID. Where `session-digest.jsonl`'s `rework` counts show outcomes without causes, join on `session_id` (when present on both streams) to attribute friction to a specific hook/rule instead of reasoning from counts alone. - `metrics/eval-ablation.jsonl` — **causal** per-agent ablation evidence from `/agent-eval --ablation <agent>` (#868): a controlled baseline-vs-ablated integration-tier delta, not accumulated usage data. When a record exists for a drop-candidate agent, Step 3 cites its measured delta/verdict instead of relying on `review-value.jsonl` alone. If no metrics data exists or insufficient data is available (fewer than 10 review runs logged), report: > "Insufficient metrics data for a meaningful audit. Run the system for a period to accumulate review data, then re-run `/harness-audit`. Minimum: 10 logged review runs." List what data is missing and exit. ### 2. Check for a stale baseline (re-baseline detection, #860) Report-only — this step never edits `evals/baseline.json` or re-runs evals itself; it only decides whether the report needs a **Re-baseline Required** section. 1. Read `evals/baseline.json`. If the file is absent, skip this step entirely (nothing to compare). 2. Read its `model` field (written by `scripts/eval_variance.py --write-baseline --model <name>`, per the change-contract flow in `skills/feedback-learning/SKILL.md`). **Absent field = pre-migration baseline — do not prompt.** This is deliberate: a baseline recorded before the `model` field existed carries no false signal either way. 3. Read the current session's model from `metrics/session-digest.jsonl` (the most recent record's model field) or session metadata. 4. Compare. **On mismatch**, the report (Step 8) gains a **Re-baseline Required** section instructing the operator to re-run the eval suite and re-write the baseline (`/agent-eval` full suite + `eval_variance.py --write-baseline --model <current-model>`) before trusting any pre/post comparison elsewhere in this report or in a feedback-learning change contract. Flag explicitly that scaffolding kept alive by old-model scores (e.g. a removal candidate from Step 3 that "still fails" on the old model) may now be re-evaluable and possibly removable. 5. **On match** (or the field absent), no section is added — this is silent success, not a finding. ### 3. Analyze review agent effectiveness For each review agent in the registry (`knowledge/agent-registry.md`): 1. **Finding rate**: How often does this agent produce findings (fail or warn) vs. pass? 2. **Zero-fail agents**: Flag agents that have never returned `fail` across all logged reviews. These are removal candidates — they may not be catching real issues. 3. **False positive rate**: If correction data exists (from `/apply-fixes`), check how often findings were dismissed vs. applied. Agents with >50% dismissed findings have a high false positive rate. 4. **Finding severity distribution**: Is the agent producing mostly minor findings? If >80% of findings are minor severity, consider whether the agent justifies its token cost. Compute this from the `severity_breakdown` object on `metrics/review-value.jsonl` rows (`{errors, warnings, suggestions}`, added in #1256) — aggregate per `agents_run` and treat `suggestions` as the minor bucket. Rows written before #1256 lack the field; count them as "no severity data" and exclude them from the ratio rather than assuming a mix (small-N honesty, consistent with Step 5). If **no** row carries `severity_breakdown`, report this analysis as dark ("severity breakdown unavailable — pre-#1256 metrics") rather than fabricating a distribution. ```bash log=".claude/metrics/review-value.jsonl"; [ -f "$log" ] || log="metrics/review-value.jsonl" [ -f "$log" ] && jq -s ' map(select(.severity_breakdown != null)) | group_by(.agents_run | sort | join(",")) | map({ agents: (.[0].agents_run | sort | join(", ")), errors: (map(.severity_breakdown.errors) | add // 0), warnings: (map(.severity_breakdown.warnings) | add // 0), suggestions: (map(.severity_breakdown.suggestions) | add // 0) }) | map(. + {total: (.errors + .warnings + .suggestions)}) | map(select(.total > 0) | . + {minor_pct: (.suggestions / .total * 100 | round)})' \ "$log" ``` ### 4. Analyze review-value fix rates **Check the sample's validity first (#2019).** Run this before reading a single rate, and report its verdict at the top of every section that cites per-lens value: ```bash python3 "$CLAUDE_PLUGIN_ROOT/skills/code-review/scripts/review_value_coverage.py" --json ``` Both writers of `review-value.jsonl` are triggered by agent instruction, not by mechanism, so the rows collected skew toward rounds that found something — an agent that found nothing is markedly less likely to run a "record the review value" step. #1512 measured this directly: ~100% "found something" across a 10-record sample, which nearly justified pruning lenses that were working fine. The script reconciles the value rows against `agent_dispatch_ledger`'s `boundary-events.jsonl` records, which a hook writes whether or not a round found anything. A verdict other than `usable` means the numbers below describe the *collection*, not the lenses: | verdict | what it means for this section | |---|---| | `no-data` | no rows; report the analysis as dark | | `unverifiable` | no dispatch records, so completeness cannot be checked | | `undercollected` | rows are a non-random subset; rates are not attributable to lenses | | `insufficient` | below the 100-row floor #1512 set | | `biased` | quiet rounds are not reaching the log; no-op rates are unreadable | | `usable` | rates may be cited | **Never emit a drop candidate, tier-down, or gating recommendation from a non-`usable` sample.** Report the verdict and what would make the sample usable instead. This is the check that would have stopped #1512's ten biased records from being read as evidence. Read `metrics/review-value.jsonl` (written by `/build` per #348, schema in `performance-metrics`). If the file is absent, note it and continue — this section is skippable. For each **checkpoint type** (the `checkpoint` field: `step` or `slice`) and each **agent combination** (`agents_run` list, treated as a set-key), compute: **Exclude read-only rows first (#1257).** Fix-rate ROI is only meaningful for fix-applying `/build` checkpoints. A read-only review (`source: "code-review"`) never applies fixes, so **every** row it produces has `issues_fixed: 0` and a 0% fix rate — feeding those to the drop-candidate logic falsely flags a whole panel that may be surfacing real defects (the 2026-07-20 run mislabeled all 7 agents this way). The `jq` filters to the **fix-applying** sources — `build-checkpoint` and, since #1962, `build-backstop` (both run the review-fix loop and can move `issues_fixed`), treating an absent `source` as `build-checkpoint` for back-compat — and drops `outcome: "skipped"` rows (a backstop suppressed by `--backstop-review=skip` never ran, so counting it would dilute every rate with a non-event), **before** grouping: ```bash log=".claude/metrics/review-value.jsonl"; [ -f "$log" ] || log="metrics/review-value.jsonl" [ -f "$log" ] && jq -s ' map(select((.source // "build-checkpoint") | . == "build-checkpoint" or . == "build-backstop")) | map(select(.outcome != "skipped")) | group_by(.checkpoint + "|" + (.agents_run | sort | join(","))) | map({ checkpoint: .[0].checkpoint, agents: (.[0].agents_run | sort | join(", ")), total: length, no_op: (map(select(.outcome=="no-op")) | length), fixed: (map(select(.outcome=="fixed")) | length), escalated: (map(select(.outcome=="escalated")) | length), fix_rate: ((map(select(.outcome=="fixed")) | length) / length * 100 | round), issues_found: (map(.issues_found) | add // 0), issues_fixed: (map(.issues_fixed) | add // 0), fix_iterations:(map(.fix_iterations)| add // 0) })' \ "$log" ``` #### Per-lens outcomes split by diff shape — the measurement a test-only gate waits on (#1964) `/code-review`'s existing cost gates narrow by file *type* (change-shape), diff *size* (change-size), and architectural *signal* (change-impact). None of them exploits a fourth, structurally-guaranteed shape: under `/test-improve`'s default `refactor-mode: no-refactor`, Phase 5's diff **cannot** contain production code, because `/build` rejects it — yet the four opus-tier `Scope: always` lenses run on it anyway, per Story and again at phase end. Whether any of them can be dropped there is an empirical question, so answer it before touching a gate. Group `diff_shape: "test-only"` rows by lens and report the outcome split against the same lens's `mixed` rows: ```bash log=".claude/metrics/review-value.jsonl"; [ -f "$log" ] || log="metrics/review-value.jsonl" [ -f "$log" ] && jq -s ' map(select((.source // "build-checkpoint") | . == "build-checkpoint" or . == "build-backstop")) | map(select(.outcome != "skipped" and .diff_shape != null)) | map({shape: .diff_shape, outcome, lens: .agents_run[]}) | group_by(.lens) | map({lens: .[0].lens, test_only: (map(select(.shape=="test-only")) | length), test_only_no_op: (map(select(.shape=="test-only" and .outcome=="no-op")) | length), mixed: (map(select(.shape=="mixed")) | length), mixed_no_op:(map(select(.shape=="mixed" and .outcome=="no-op")) | length)}) | sort_by(-.test_only)' \ "$log" ``` Report `test_only_no_op / test_only` per lens, alongside that lens's `mixed` rate as the control — a lens that no-ops at the same rate on *both* shapes is simply a quiet lens, not one this diff shape defeats, and gating it on shape would be reading noise as signal. Only a lens that no-ops on test-only diffs **and** earns its keep on mixed ones is a candidate for `change_shape.py`'s `TEST_ONLY_SKIP_LENSES`, and each entry lands in its own PR citing these numbers. State the row count: with few rows, the honest finding is "not enough data yet", not a recommendation. Two lenses are **not** candidates regardless of what the split shows, and the report should say so rather than proposing them: `security-review` (tests routinely embed credentials and injection payloads) and `correctness-review` (an inverted assertion is exactly its subject). #### Backstop redundancy — the measurement that gates `--backstop-review=skip` (#1962) `/build`'s Step-6 backstop reviews files an inline checkpoint (sub-steps 4/6) already reviewed in the same run, and under an enclosing orchestrator (e.g. `/test-improve` Phase 5, which runs its own end-of-phase panel over the cumulative diff) it is the third review layer over the same test code. Whether that layer earns its cost is an empirical question, and `source: "build-backstop"` rows are the answer. Report the backstop's own outcome split next to the checkpoint split, restricted to builds where a checkpoint actually ran first — a backstop on a `trivial`-only slice reviewed something nothing else did, and including it would understate redundancy: