risk-tiered-validation-selector

name

risk-tiered-validation-selector

description

Machine-classify a change's touched files into impact classes that select validation DEPTH — docs-only fast path, fast tier, or full tier — with a classifier that FAILS CLOSED: unmatched, ambiguous, or error cases escalate to full, never down. Ships tier check bundles, an explicit docs-only definition PLUS a "never docs-only" list (workflows, scripts, schema/migrations, auth/RLS, lockfiles, agent-instruction files), forced-full lists for high-risk surfaces, and diffable rules; aggregation is max-over-files, no averaging. Routes validation COST where change-classification-gate routes APPROVAL — compose, don't merge. Use when picking fast vs full validation for a change, designing tiered validation, when docs-only changes waste full CI, or when risky files sneak through light checks. Do NOT use to set approval paths or scope lock (change-classification-gate), to run the selected tier (local-ci-mirror-preflight executes it), or to architect the full tier's shards (sharded-validation-with-resume).

Risk-Tiered Validation Selector

Purpose

Spend validation where risk lives — mechanically, and erring expensive. Flat validation policy fails in both directions: full CI on a typo teaches people to resent (and bypass) validation, while light checks on a migration is how incidents start. The evidence pattern (Repo B's impact classifier + tier protocol, corroborated by Repo A's docs-only rule) is a CLASSIFIER whose input is the changed file set and whose output is a validation tier — with the defining property that it FAILS CLOSED: a file no rule matches, an ambiguous match, or a classifier error selects FULL validation. Cheap tiers are earned by explicit rules; expensive is the default for the unknown. This skill designs that classifier and its tier definitions as reviewable repo artifacts.

Use When

• Use when: deciding, for a concrete change, whether the docs-only path, fast tier, or full tier applies (running the classification).
• Use when: designing or overhauling a repo's tiered-validation scheme — the tiers, the rules, the lists.
• Use when: docs-only changes are burning full-CI minutes, or risky files keep riding through light validation.
• Use when: the docs-only definition is contested ("it's just a config tweak") and needs a written boundary.
• Do NOT use when: deciding approval paths, human sign-off, or scope lock — that is change-classification-gate; it routes changes to APPROVAL and this selector routes them to validation COST. A change gets both, independently.
• Do NOT use when: actually executing the selected tier before a commit — that is local-ci-mirror-preflight (it mirrors and runs; this skill only selects).
• Do NOT use when: the full tier's internals (shards, resume, aggregate gate) are the subject — that is sharded-validation-with-resume.
• Do NOT use when: adding the classifier job to CI — the wiring composes ci-pipeline-architect.

Inputs to Inspect

1. The changed file set (for a live classification): git diff --name-only against the PR base — the classifier's only input is files, never the author's description of them.
2. The repo's risk topology (for design): where schema/migrations, auth/RLS, payment paths, CI workflows, agent-instruction files, generated code, and pure-docs trees live — these seed the lists.
3. The existing validation inventory: what checks exist and their costs/ durations — tiers are bundles of real checks, not aspirations.
4. change-classification-gate's class definitions where present — the two taxonomies should share vocabulary (a schema-migration class maps to forced-full) without merging responsibilities.
5. History of misroutes: past incidents where a light-validated change broke something (candidates for forced-full), and heavyweight runs on trivial diffs (candidates for explicit fast rules).

Workflow

1. Define the tiers as check bundles: docs-only (reference/link sanity
   • docs-scoped jobs), fast (typecheck/lint/build + focused unit scope), full (everything, incl. the sharded suite where sharded-validation-with-resume exists). Name real commands/jobs per tier — a tier without a check list is a mood.
2. Write the docs-only definition — positively and negatively. Positive: which paths/extensions are documentation. Negative (the "never docs-only" list): CI workflows, scripts, schema/migrations, auth/RLS/policy paths, lockfiles/dependency manifests, agent-instruction files, generated code — matching ANY of these disqualifies the whole change from docs-only regardless of what else it touches.
3. Write the forced-full list: surfaces whose risk ignores diff size — schema/migrations, auth/RLS, payment/billing, tenant isolation, CI/CD definitions, the classifier's own rules file. Any hit → full, no averaging.
4. Write the classification rules as a reviewable artifact (rule table or script design — format: references/classifier-rules.md): ordered glob → class mappings, most-specific first, with the LAST rule being the fail-closed catch-all: * → full (unmatched).
5. Fix the aggregation rule: a change's tier is the MAX tier over its files (docs-only < fast < full). One workflow file in a 40-file docs diff makes the change full. No file-count thresholds, no averaging.
6. Fail closed on classifier errors too: diff unavailable, rules file unparseable, glob engine error → full. The classifier must never convert its own failure into a cheap tier.
7. For a live classification, emit the record: files → matched rule → per-file class → aggregate tier → the checks that tier requires (which local-ci-mirror-preflight then mirrors). Unmatched files are NAMED in the record — each is a rules-file gap to close, not an annoyance to suppress.
8. Keep the rules honest over time: every misroute (light-validated breakage) adds a rule or a forced-full entry; the rules file is change-controlled like the code it guards (and edits to it are themselves forced-full).

Output Format

VALIDATION TIER SELECTION — <branch/PR> (<n> files vs <base>)
Per-file:
  | file | matched rule | class |
  | docs/guide.md | docs/** → docs-only | docs-only |
  | .github/workflows/ci.yml | never-docs-only list | full (forced) |
  | src/lib/util.ts | src/** → fast | fast |
  | tools/gen/output.g.ts | (no rule matched) | full (fail-closed) |
Aggregate tier: FULL (max over files) — because: <the deciding file(s)>
Tier contents: <the named checks/jobs this tier runs>
Rules-file gaps: <unmatched files to add rules for>

Validation Checklist

• Every tier names its actual checks/commands; docs-only has BOTH a positive definition and a never-docs-only list.
• The rules artifact ends in the fail-closed catch-all; classifier errors also route to full.
• Aggregation is max-over-files; no rule averages, counts files, or trusts the PR description.
• Forced-full list covers the repo's genuinely high-risk surfaces, including the classifier's own rules file and CI definitions.
• For live runs: every file shows its matched rule; unmatched files are named as gaps.
• The selector's vocabulary aligns with change-classification-gate's classes without absorbing its approval/scope duties.
• The rules are a diffable repo artifact, not prose folklore.

Gotchas

• Fail-open by generosity: "it's obviously harmless, call it fast" for an unmatched file — the entire design collapses at the first exception. Unmatched means full; then ADD THE RULE so next time matches.
• The renamed/moved file: renames of risky paths must classify by BOTH old and new path (a migration moved is still a migration).
• Deleted files count: a deletion under a forced-full path is a forced-full change; classifiers that only look at additions miss it.
• Generated-code blind spot: generated output looks harmless, but the generator input that produced it may not be in the diff; classify generated trees explicitly (usually fast-or-full by what consumes them), never by "it's autogenerated, skip it".
• Description trust: classifying from the PR title/author claim instead of the file list re-opens the exact hole the machine classification closes.
• Tier inflation: if everything ends up full, the fast rules are too thin and the selector dies of uselessness — mine the full-tier runs on genuinely trivial diffs for safe fast rules. Fail-closed does not mean never-fast; it means EARNED-fast.

Stop Conditions

• Asked to classify an unmatched or never-docs-only file into a cheaper tier "just this once" → refuse; fail-closed is the design's load-bearing property. The lawful path is a reviewed rules-file change.
• Asked to design tiers with no real check inventory to bundle (nothing to select between) → stop; the validation checks must exist first (or route to designing them).
• The diff/base cannot be computed (shallow clone, missing base) → the classification is FULL by fail-closed error handling; say so rather than guessing from filenames at hand.
• Asked to fold approval decisions into the tier output ("full tier implies human sign-off, so skip the gate") → refuse; approval routing stays with change-classification-gate — shared vocabulary, separate authorities.

Supporting Files

• references/classifier-rules.md — the rules-artifact format (ordered globs, forced-full, never-docs-only, catch-all), starter lists, aggregation and error-handling semantics, and a worked classification.
• evals/evals.json — behavior cases incl. fail-closed on unmatched files and mixed docs+workflow diffs.
• evals/trigger-evals.json — discrimination against change-classification-gate, local-ci-mirror-preflight, and sharded-validation-with-resume.