validate-findings

# Validate Findings — Live Validation & Attack-Chain Harness > **Paths.** `analysis-results/…` and `progress-tracker/…` in this skill are the > default workspace layout. They resolve through `locations.yaml` in > `$TRAUST_CONFIG_HOME` (`docs/setup.md`, Storage locations); substitute your > configured roots. Take the static outputs of `secure-code-audit`, `threat-model`, and `triage` and **prove or refute them against a live authorized environment**. Then go further: chain confirmed findings into multi-step kill-chains and hunt for novel attacks the static analysis missed. > **Authorization**: This skill executes potentially state-changing actions against live infrastructure. It MUST only be run against targets you own or have explicit written authorization to test. Every action is gated by the scope guard (`scope.py`) — out-of-scope steps are refused and logged, never silently skipped. --- ## Input `$ARGUMENTS` is parsed as whitespace-separated tokens. The **first non-flag token** is the findings source; remaining tokens are flags. ### Findings source (required — one of) | Form | Resolution | |---|---| | `<dir>/` | Directory containing `*-security-audit.{json,md}`, `*-threat-model.md`, `*-triage.{json,md}` | | `<file>.{md,json}` | A single report file; siblings auto-discovered in the same directory | | `<product>/<repo>` | Shorthand resolved to `../analysis-results/findings/<product>/<repo>/` | | `<slug>-findings` | Product package from `../progress-tracker/processed-results/<slug>-findings/`. All repos in the package are ingested into a single model; each finding is tagged with its source repo. The bare `<slug>` form (without `-findings`) is also accepted. | | `<slug1>,<slug2>,…` | **Multiple packages** merged into one model — used when a logical product spans several `processed-results` entries (see the Deduplication Map in `progress-tracker/VALIDATION-PRIORITY-LIST.md`, e.g. ODF → 12 packages). Findings are tagged `<package>:<repo>` and de-duplicated by `(repo, id)` so a repo appearing in two packages contributes once. | | `all-confirmed` | Every triage-confirmed finding across `../analysis-results/findings/**` (use with `--dry-run` first) | **Refuted findings stay in scope.** When a `<repo>-refuted-register.json` sits next to the audit report (emitted by python3 -m traust.cli ledger emit-triage), its entries are candidate targets, not exclusions: a finding dismissed as a false positive — even human-countersigned — is a falsifiable claim, and a reproducing exploit from this skill overrides the assertion via evidence-class precedence (`fp_overridden`, loudly surfaced with the countersigner attributed). Include register entries in attack planning when scope and budget allow. **Impact-analysis artifacts as target selection.** When `analysis-results/impact/<cve>-impact-analysis.json` (from `/impact-analysis`) covers the finding's CVE, use it two ways: (a) prioritization — repos classified `affected` with triage's `attacker_influence: plausible` are the validation candidates worth cluster time first; (b) attack planning — the repo's `evidence.govulncheck_trace` names the call path caller-first, so the first in-repo frame identifies which interface the attack plan should drive. A confirmed exploitation here is execution evidence (class 1): it flips the finding to `confirmed` in the disposition ledger and is the terminal answer to "is this repo actually affected". **Product packages** (`*-findings/` directories) contain sub-directories grouped by repo or by sub-group/repo (1–2 levels). The harness auto-detects the package layout: if the source directory has no audit reports at the top level but has sub-directories with audit reports, it is treated as a package and all repos are ingested. Findings from each repo carry a `source_repo` tag for traceability in the plan, execution log, and validation report. ### Scope binding (at least one mode; modes are additive) | Flag | Mode | Effect | |---|---|---| | `--targets <file.yaml>` | **1 — explicit** | Load a rules-of-engagement scope file (see `targets.example.yaml`). Highest precedence. | | `--context <name>` | **2 — inline** | Kubeconfig context to bind (repeatable). Implies the cluster at that context is in scope. | | `--ns <name>` | 2 | Namespace allowlist (repeatable, glob OK). | | `--image <ref>` | 2 | Container image allowlist (repeatable, glob OK). | | `--pod <selector>` | 2 | Pod label selector allowlist (repeatable). | | `--container <name>` | 2 | Running container name/ID for the container adapter (repeatable, glob OK). | | `--wasm <path>` | 2 | WASM artifact path for the wasm adapter (repeatable). | | `--infer-scope` | **3 — inferred** | Derive scope from report `metadata` + threat-model `entry_points`. Lowest precedence; **never** infers `kube-system`, `openshift-*` control-plane, or `default` namespaces. | If no scope mode is given, the harness runs `--dry-run` implicitly and warns. ### Execution control | Flag | Effect | |---|---| | `--dry-run` | Stop after Phase 2. Emit `attack-plan.yaml` only; nothing touches the target. | | `--auto` | Skip the Phase 3 review gate. Use **only** in isolated lab environments. | | `--replay-only` | Validate existing findings only; skip chaining and novel hunting. | | `--novel-only` | Skip replay; run recon + chain synthesis + novel probes only. | | `--destructive` | Permit steps the adapter classifies as `destructive` (data loss, DoS, irreversible mutation). Without this flag such steps are recorded as `not_attempted` with reason `destructive-not-permitted`. | | `--max-novel <N>` | Cap novel-attack hypotheses (default 10). | | `--out <dir>` | Override output directory (default: alongside the source reports). | --- ## Scope Binding — Resolution & Enforcement Run `python harnessing/5-validate/validate-findings/scope.py` semantics: 1. **Load** mode-1 file if `--targets` given; merge mode-2 inline flags on top; if `--infer-scope`, call `ingest.infer_scope()` and merge with **lowest** precedence. 2. **Compile** into a `Scope` object exposing `is_in_scope(action: Action) -> (bool, reason)` where `Action = {adapter, verb, context, namespace, resource, name, image, extra}`. 3. **Hard denies** (always refused regardless of flags or `--auto --destructive`): - Any entry in `targets.yaml#off_limits`. - Any namespace matching `kube-system`, `openshift-etcd`, `openshift-kube-apiserver*`, `openshift-authentication*` **unless explicitly listed** in mode-1 `clusters[].namespaces`. - `expires` date in the past. 4. **Preflight** every bound target via the adapter's `preflight()` to capture a fingerprint (cluster version, node count, image digest, WASM sha256). The fingerprint goes into `validation.json#metadata.target_fingerprint` so results are reproducible. Every executed step is logged with the scope-check outcome. A `blocked_by_scope` verdict is a **result**, not an error — it tells the reader the PoC would have crossed a boundary the engagement does not permit. --- ## Benchmark mode (P7) `analysis-results/scan-testing/validation-benchmark/benchmark-findings.json` (from python3 -m traust.cli sweep benchmark plan) is a first-class input: validate its claims against the deployed fixtures exactly like campaign findings, then score the report with `run_validation_benchmark.py score --variant vuln|safe`. Benchmark runs live in the harness-QA tree and never touch campaign metrics. ## Phase 0b — Target attestation (fail-closed, P2) After scope loads and before ANY probe executes, attest the target and write `target-attestation.json` into the validation output dir: ```bash python3 -m traust.cli admin attest-target --out <out-dir>/target-attestation.json \ --namespace <operand-ns> [--csv <operator-csv>] [--selector <pods>] \ [--version <deployed> --affected-range '<vX.Y.Z'] \ [--kubeconfig <path>] [--context <ctx>] ``` Record the summary in `metadata.target_attestation`. If `attested: false`: probes MAY still run for diagnosis, but every verdict in the report is structurally void — `emit_validation_ledger_events` routes the entire run to `needs_review` as `environment_invalid` (or `attestation_missing` for post-0.176.0 reports without the artifact). The rhoso.v2 failure class — 34 false "refuted" from a never-installed operator — becomes 0 verdicts at the front door. ## Phase 0c — Positive-control pairing (assay validity, P1) Every **refutation-capable** probe step carries paired positive controls in its plan entry, executed in the same session and recorded in the step's `controls[]` (schema `positive_control`): | Probe class | Paired control (kind) | |---|---| | RBAC "subject X cannot do Y" | same client performs an action X is KNOWN to be allowed (`must_succeed`) — proves auth worked, enumeration non-empty, API reachable | | Secret-exposure "secret not readable" | read a planted canary secret the probe SHOULD see (`must_succeed`) — proves the oracle observes the right store | | Network/exposure "endpoint not reachable" | reach a known-open endpoint on the same path (`must_succeed`) and a known-closed one (`must_deny`) | Rules (enforced by the soundness gate + ledger emitter): - A `refuted` verdict with a **failed** control is quarantined as `failed-positive-control` — any report age: the assay demonstrably didn't work. - A `refuted` verdict with **no passing control**, on reports at/after harness 0.177.0, quarantines as `missing-positive-control`. - Controls never substitute for the probe: they prove the assay could have detected the claim, converting "absence of evidence" into "evidence of absence, with the assay proven live." ## Phase 0d — Differential probing for authz claims (P3) For every "role/subject can(not) do X" refutation, probe the **pair** in the same session and record it in the step's `differential` block: - the claimed action, AND - a neighbor action with a **known-different expected outcome** (same subject → a verb it IS allowed; same verb → a subject that IS authorized). If both outcomes are identical, the oracle cannot discriminate allowed from denied — the refutation is unsound **regardless of which way it pointed** (`non-discriminating-oracle`, quarantined at any report age). Post-0.178.0, an authz-class refutation without a discriminating differential quarantines as `missing-differential-probe`. One extra request per probe; catches wrong-oracle failures without knowing *why* the oracle is wrong. ## Phase 0e — Severity validation on confirmations (P9) Every `confirmed` finding records `severity_validation`: the CVSS components the exploit ACTUALLY demonstrated (attack vector used, privileges the probe identity held at success, user interaction, scope crossing observed from the E0 artifact, impact axes evidenced) and the signed delta vs the claimed score. When |delta| >= 1.0 **and** the evidence grade is E0/E1/E2 (never E3), emit a `proposal` — the emitter routes it to the countersign severity decision as a needs_review item; machines never write `disposition.severity`. A demonstrated DOWNGRADE ("exploitation required cluster-admin") is exactly as valuable as an upgrade. Findings that are constituents of a demonstrated attack chain also record `chain_context` (chain id, chain severity, role) — chain membership is severity evidence and belongs in the proposal rationale. ## Phase 6 pointer — discovery sweeps & replay artifacts (P5/P8) **Replay (P8, required on every probe):** record a self-contained replay script per executed step in `artifacts/replay/<step_id>.sh` (+ inputs file when the probe posts data) and reference it in the step's `replay` block with the attestation fingerprint sha. A countersign human re-runs the exact probe instead of trusting the transcript. **Discovery sweeps (P5, optional phase after replay/chained/novel):** outputs are NEW finding candidates with `origin: validation-discovery`, routed to `/triage` generic ingest — never straight to the ledger: 1. *State diffing (implemented):* python3 -m traust.cli impact cluster-state-diff snapshot before the sweep and after it; `diff` turns unexpected deltas (new/mutated RBAC, SCC changes, webhook mutations, deleted NetworkPolicies, new exposure) into candidates, excluding declared probe side-effects (`--expected`). 2. *Anonymous-surface sweep* — unauthenticated reachability vs an authenticated baseline (per-sweep increment, not yet implemented). 3. *Privilege-escalation chain search* — seeded from operator-priv-profile + the RBAC graph; attempt the cheapest link under the scope guard (not yet implemented). 4. *Browser role×route matrix + IDOR probing* (browser lane; not yet implemented). All sweeps inherit the hard scope guard unchanged. ## Phase 1 — Ingest & Normalize ```bash python harnessing/5-validate/validate-findings/ingest.py <findings-source> > /tmp/normalized.json ``` 1. Locate the three report types in the source directory. JSON is preferred; fall back to Markdown parsing when JSON is absent. 2. Build a unified `Finding` list. For each finding merge: - `security-audit`: `id`, `title`, `severity`, `cwes`, `locations`, `evidence[]`, `attack_pattern`, `cvss` - `triage` (if present): `verdict`, `verify_verdict`, `confidence`, `preconditions[]`, `first_links[]`, `severity_label`, `owner_hint` - `threat-model` (if present): linked `threat_id`, `entry_point`, `asset`, `controls` 3. Build a `ThreatGraph` skeleton: assets (with sensitivity), entry points (with trust level + reachable assets), threats (with status + linked finding evidence). 4. Extract **embedded PoCs**: scan `evidence[]` code blocks and `attack_pattern` text for fenced `yaml`/`bash`/`json`/`curl` blocks and the triage `rationale` STEP narratives. Tag each as `{lang, body, source_field}`. Findings with triage `verdict == false_positive` are carried through but default to `technique: skip` in the plan (override with `--include-fp`). Findings with triage `verdict == needs_review` (evidence not statically locatable; never proven false) are **not** skipped — they are planned like `true_positive` and the resulting step is tagged `triage_was_needs_review: true`. --- ## Phase 2 — Attack Planning ```bash python harnessing/5-validate/validate-findings/plan.py /tmp/normalized.json --scope <scope.json> > <out>/<repo>-attack-plan.yaml ``` The plan is an ordered list of **steps**, each: ```yaml - id: step-007 finding_ref: ODR-2026-004 # or novel_ref / chain_ref technique: replay # replay | adapted | chained | novel | recon adapter: k8s verb: apply-manifest target: {context: lab-spoke-1, namespace: ramen-ops} payload: | # the literal manifest / command / wasm-invoke apiVersion: ramen.openshift.io/v1alpha1 kind: Recipe ... classification: mutating # safe | mutating | destructive (from adapter.classify) expected: "pods/exec succeeds in namespace outside tenant scope" rollback: "kubectl --context lab-spoke-1 -n ramen-ops delete recipe pwn" preconditions: [step-003] # steps that must succeed first ``` ### 2a. Replay steps For each non-FP finding, convert extracted PoCs into one or more steps: | PoC form | Adapter / verb | |---|---| | YAML manifest (Kind present) | `k8s / apply-manifest` | | `kubectl …` / `oc …` literal | `k8s / raw` | | `curl` / `http` against a Service | `k8s / port-forward+http` (or `container / network-probe` if no cluster) | | `podman exec` / `docker exec` | `container / exec` |