validate-findings

name

validate-findings

description

Use when the user asks to validate, reproduce, exploit, red-team, or live-test security findings against a Kubernetes/OpenShift cluster, operator, WASM module, container, pod, or cluster component. Ingests *-security-audit, *-threat-model, and *-triage reports, builds an attack plan (replay + chained + novel), executes it against an authorized live target under a hard scope guard, and emits a schema-validated *-validation.{json,md} report.

Validate Findings — Live Validation & Attack-Chain Harness

Take the static outputs of secure-code-audit, threat-model, and triage and prove or refute them against a live authorized environment. Then go further: chain confirmed findings into multi-step kill-chains and hunt for novel attacks the static analysis missed.

Authorization: This skill executes potentially state-changing actions against live infrastructure. It MUST only be run against targets you own or have explicit written authorization to test. Every action is gated by the scope guard (scope.py) — out-of-scope steps are refused and logged, never silently skipped.

---

Input

$ARGUMENTS is parsed as whitespace-separated tokens. The first non-flag token is the findings source; remaining tokens are flags.

Findings source (required — one of)

Form	Resolution
<dir>/	Directory containing *-security-audit.{json,md}, *-threat-model.md, *-triage.{json,md}
<file>.{md,json}	A single report file; siblings auto-discovered in the same directory
<product>/<repo>	Shorthand resolved to ../analysis-results/findings/<product>/<repo>/
<slug>-findings	Product package from ../progress-tracker/processed-results/<slug>-findings/. All repos in the package are ingested into a single model; each finding is tagged with its source repo. The bare <slug> form (without -findings) is also accepted.
<slug1>,<slug2>,…	Multiple packages merged into one model — used when a logical product spans several processed-results entries (see the Deduplication Map in progress-tracker/VALIDATION-PRIORITY-LIST.md, e.g. ODF → 12 packages). Findings are tagged <package>:<repo> and de-duplicated by (repo, id) so a repo appearing in two packages contributes once.
all-confirmed	Every triage-confirmed finding across ../analysis-results/findings/** (use with --dry-run first)

Refuted findings stay in scope. When a <repo>-refuted-register.json sits next to the audit report (emitted by scripts/emit_triage_ledger_events.py), its entries are candidate targets, not exclusions: a finding dismissed as a false positive — even human-countersigned — is a falsifiable claim, and a reproducing exploit from this skill overrides the assertion via evidence-class precedence (fp_overridden, loudly surfaced with the countersigner attributed). Include register entries in attack planning when scope and budget allow.

Product packages (*-findings/ directories) contain sub-directories grouped by repo or by sub-group/repo (1–2 levels). The harness auto-detects the package layout: if the source directory has no audit reports at the top level but has sub-directories with audit reports, it is treated as a package and all repos are ingested. Findings from each repo carry a source_repo tag for traceability in the plan, execution log, and validation report.

Scope binding (at least one mode; modes are additive)

Flag	Mode	Effect
--targets <file.yaml>	1 — explicit	Load a rules-of-engagement scope file (see targets.example.yaml). Highest precedence.
--context <name>	2 — inline	Kubeconfig context to bind (repeatable). Implies the cluster at that context is in scope.
--ns <name>	2	Namespace allowlist (repeatable, glob OK).
--image <ref>	2	Container image allowlist (repeatable, glob OK).
--pod <selector>	2	Pod label selector allowlist (repeatable).
--container <name>	2	Running container name/ID for the container adapter (repeatable, glob OK).
--wasm <path>	2	WASM artifact path for the wasm adapter (repeatable).
--infer-scope	3 — inferred	Derive scope from report metadata + threat-model entry_points. Lowest precedence; never infers kube-system, openshift-* control-plane, or default namespaces.

If no scope mode is given, the harness runs --dry-run implicitly and warns.

Execution control

Flag	Effect
--dry-run	Stop after Phase 2. Emit attack-plan.yaml only; nothing touches the target.
--auto	Skip the Phase 3 review gate. Use only in isolated lab environments.
--replay-only	Validate existing findings only; skip chaining and novel hunting.
--novel-only	Skip replay; run recon + chain synthesis + novel probes only.
--destructive	Permit steps the adapter classifies as destructive (data loss, DoS, irreversible mutation). Without this flag such steps are recorded as not_attempted with reason destructive-not-permitted.
--max-novel <N>	Cap novel-attack hypotheses (default 10).
--out <dir>	Override output directory (default: alongside the source reports).

---

Scope Binding — Resolution & Enforcement

Run python harnessing/validate-findings/scope.py semantics:

1. Load mode-1 file if --targets given; merge mode-2 inline flags on top; if --infer-scope, call ingest.infer_scope() and merge with lowest precedence.
2. Compile into a Scope object exposing is_in_scope(action: Action) -> (bool, reason) where Action = {adapter, verb, context, namespace, resource, name, image, extra}.
3. Hard denies (always refused regardless of flags or --auto --destructive):
   • Any entry in targets.yaml#off_limits.
   • Any namespace matching kube-system, openshift-etcd, openshift-kube-apiserver*, openshift-authentication* unless explicitly listed in mode-1 clusters[].namespaces.
   • expires date in the past.
4. Preflight every bound target via the adapter's preflight() to capture a fingerprint (cluster version, node count, image digest, WASM sha256). The fingerprint goes into validation.json#metadata.target_fingerprint so results are reproducible.

Every executed step is logged with the scope-check outcome. A blocked_by_scope verdict is a result, not an error — it tells the reader the PoC would have crossed a boundary the engagement does not permit.

---

Phase 1 — Ingest & Normalize

python harnessing/validate-findings/ingest.py <findings-source> > /tmp/normalized.json

1. Locate the three report types in the source directory. JSON is preferred; fall back to Markdown parsing when JSON is absent.
2. Build a unified Finding list. For each finding merge:
   • security-audit: id, title, severity, cwes, locations, evidence[], attack_pattern, cvss
   • triage (if present): verdict, verify_verdict, confidence, preconditions[], first_links[], severity_label, owner_hint
   • threat-model (if present): linked threat_id, entry_point, asset, controls
3. Build a ThreatGraph skeleton: assets (with sensitivity), entry points (with trust level + reachable assets), threats (with status + linked finding evidence).
4. Extract embedded PoCs: scan evidence[] code blocks and attack_pattern text for fenced yaml/bash/json/curl blocks and the triage rationale STEP narratives. Tag each as {lang, body, source_field}.

Findings with triage verdict == false_positive are carried through but default to technique: skip in the plan (override with --include-fp). Findings with triage verdict == needs_review (evidence not statically locatable; never proven false) are not skipped — they are planned like true_positive and the resulting step is tagged triage_was_needs_review: true.

---

Phase 2 — Attack Planning

python harnessing/validate-findings/plan.py /tmp/normalized.json --scope <scope.json> > <out>/<repo>-attack-plan.yaml

The plan is an ordered list of steps, each:

- id: step-007
  finding_ref: ODR-2026-004        # or novel_ref / chain_ref
  technique: replay                # replay | adapted | chained | novel | recon
  adapter: k8s
  verb: apply-manifest
  target: {context: lab-spoke-1, namespace: ramen-ops}
  payload: |                       # the literal manifest / command / wasm-invoke
    apiVersion: ramen.openshift.io/v1alpha1
    kind: Recipe
    ...
  classification: mutating         # safe | mutating | destructive (from adapter.classify)
  expected: "pods/exec succeeds in namespace outside tenant scope"
  rollback: "kubectl --context lab-spoke-1 -n ramen-ops delete recipe pwn"
  preconditions: [step-003]        # steps that must succeed first

2a. Replay steps

For each non-FP finding, convert extracted PoCs into one or more steps:

PoC form	Adapter / verb
YAML manifest (Kind present)	k8s / apply-manifest
kubectl … / oc … literal	k8s / raw
curl / http against a Service	k8s / port-forward+http (or container / network-probe if no cluster)
podman exec / docker exec	container / exec
Shell against a pod	k8s / exec
WASM invoke / hostcall	wasm / invoke-export
No literal PoC	technique: adapted — synthesize a minimal probe from cwes + locations (e.g. CWE-918 ⇒ SSRF callback to a harness-controlled listener)

2b. Chained steps (chain.py)

Build the attack graph and emit one chain_ref step-group per discovered path:

• Nodes: findings, threat-model entry points, threat-model assets, recon-discovered resources (added in Phase 4 second pass).
• Edges:
  • entry_point → finding when the finding's preconditions are satisfiable at that entry point's trust level.
  • finding_A → finding_B when A's expected impact (token theft, ns escape, RBAC grant, file write, network position) satisfies any of B's preconditions. Use the capability vocabulary in chain.py:CAPABILITIES.
  • finding → asset when the finding's impact references the asset.
• Search: DFS from every entry point to every asset with sensitivity ≥ high. Deduplicate by node-set. Cap path length at 6.
• For each chain, emit ordered steps reusing the replay steps where they exist, inserting technique: chained glue steps (e.g. "extract SA token from step-003 evidence and set KUBECONFIG for step-004").
• Annotate with MITRE ATT&CK for Containers technique IDs (chain.py:MITRE_MAP).

2c. Novel steps (novel.py)

1. Recon plan: emit technique: recon steps for the bound target type(s) — these are always classification: safe:
   • k8s/operator/component: kubectl get crd,clusterrolebinding,rolebinding,networkpolicy,validatingwebhookconfiguration,mutatingwebhookconfiguration,svc,ep -A -o json; kubectl auth can-i --list --as=<each in-scope SA>; kubectl get pods -o jsonpath for securityContext, hostPath, SA mounts, capabilities.
   • container: inspect; cat /proc/self/status (CapEff), /proc/mounts, env, listening sockets.
   • wasm: wasm-tools dump --imports --exports; component-model world inspection.
2. Diff recon output against threat-model entry_points + controls. Surfaces present live but absent from the model are candidate novel entry points.
3. Pattern probes — for each candidate, emit a probe step from the catalogue in novel.py:PROBE_CATALOGUE keyed by target type and surface class:
   • Operator: confused-deputy (cross-ns *Ref fields in CRD schema), SSRF (URL-typed CRD fields → harness listener), TOCTOU (status-subresource race), webhook bypass (label/annotation that skips admission), over-broad RBAC (auth can-i --list shows * * *).
   • Pod / container: writable hostPath, CAP_SYS_ADMIN/CAP_NET_ADMIN presence, /var/run/docker.sock or CRI socket mount, SA token with cluster-admin reachability, procfs breakout, shared PID/net namespace.
   • Cluster component: unauth metrics/pprof/healthz with side-effects, anonymous-auth on kubelet, insecure-port residue, etcd client cert reuse.
   • WASM: unrestricted WASI preopen dirs, hostcall import without capability check, linear-memory OOB on exported function, missing fuel/epoch limits.
4. AI-assisted hypotheses — after recon executes (Phase 4 first pass), reason over: recon output, unexploited threat-model threats (status: unmitigated with no linked finding), and the CWE distribution of confirmed findings. Propose up to --max-novel additional attack hypotheses. Each becomes a technique: novel step with a clear expected outcome that distinguishes confirmed from refuted.

2d. Ordering & classification

Order: all recon → all replay (severity desc) → chained → novel. Run adapter.classify() on every step; if destructive and --destructive not set, mark skip: destructive-not-permitted.

Write the plan to <out>/<repo>-attack-plan.yaml.

---

Phase 3 — Review Gate

Unless --auto or --dry-run:

1. Print a summary table: step count by {technique × classification}, every mutating/destructive step's verb + target, and every step that scope.is_in_scope() would refuse.
2. Ask the user to approve. Accept: yes (run all permitted), edit (open attack-plan.yaml, re-read, re-summarize), skip <id...> (mark listed steps not_attempted: user-skip), or no (exit, plan kept on disk).
3. Record the approval (who, when, mode) in validation.json#metadata.approval.

If --dry-run: stop here. The plan on disk is the deliverable.

---

Phase 4 — Execute

python harnessing/validate-findings/execute.py <out>/<repo>-attack-plan.yaml --scope <scope.json> --out <out>

For each step in plan order:

1. scope.is_in_scope(step.action) — if false, record verdict: blocked_by_scope with reason; continue.
2. Check preconditions — if any referenced step did not yield verdict: confirmed, record verdict: not_attempted with reason precondition-failed:<id>; continue.
3. Dispatch to TARGET_ADAPTERS[step.adapter].execute(step, scope, audit):
   • Capture stdout/stderr/exit, plus adapter-specific evidence (resource YAML before/after, HTTP response body, WASM trap).
   • Write artifacts to <out>/artifacts/step-<id>.* and compute sha256.
   • Compare observed outcome to step.expected → verdict ∈ {confirmed, refuted, inconclusive}.
4. Append one JSONL line to <out>/validation-audit.jsonl:

   {"ts":"<iso8601>","step_id":"...","adapter":"...","verb":"...","target":"...","scope_check":"pass","classification":"...","verdict":"...","evidence":["artifacts/step-007.log"],"rollback":"..."}
   

5. If classification: mutating and the adapter provides rollback(), run it immediately after evidence capture and record rollback_performed: true|false with output.
6. Two-pass for novel: after recon steps complete, re-invoke novel.py with live recon artifacts to materialize the AI-assisted hypotheses (Phase 2c-4) into concrete steps, append them to the plan, and continue execution.

Rate-limit: ≤ 1 mutating step per 2 s per cluster context; back off ×2 on 429/503 from the API server.

---

Phase 5 — Report

python harnessing/validate-findings/report.py <out> > <out>/<repo>-validation.json
python scripts/validate_report.py --schema schema/validation.schema.json <out>/<repo>-validation.json
python harnessing/validate-findings/report.py <out> --markdown > <out>/<repo>-validation.md

<repo>-validation.json conforms to schema/validation.schema.json:

• metadata: date, harness_version (VERSION + short SHA), scope_binding_mode (explicit|inline|inferred|mixed), target_fingerprint[], approval, flags.
• source_reports[]: paths + sha256 of the ingested audit/threat-model/triage files.
• summary: counts by verdict, by technique, novel-finding count, highest-impact confirmed chain.
• validated_findings[]: one per source finding — source_id, verdict, technique, evidence[], observed_impact, deviation_from_claim, rollback_performed.
• attack_chains[]: chain_id, name, entry_point, terminal_asset, mitre_attack_refs[], steps[], verdict.
• novel_findings[]: full report.schema.json#/$defs/finding objects + discovery_method + chain_context — ready to feed into file-security-defect.
• execution_log_ref: relative path to validation-audit.jsonl.
• negative_results[]: probes that ran clean (reuse report.schema.json#/$defs/negative_result).

The Markdown render mirrors the structure of *-security-audit.md: executive summary table, per-finding verdict cards with embedded evidence excerpts, an attack-chain section with ASCII path diagrams, a novel-findings section formatted identically to audit findings, and an appendix linking every artifact.

---

Safety Controls

• Default-deny scope: no action runs without a positive scope match. Inferred scope never includes control-plane namespaces.
• Staged by default: human reviews the plan unless --auto. --auto is refused if metadata.approval.environment is not lab in targets.yaml.
• Destructive opt-in: steps that delete data, kill pods, exhaust resources, or cannot be rolled back require --destructive AND must not match off_limits.
• Rollback: apply-manifest and create-cr auto-delete what they created; patch-cr captures pre-image and restores it; exec/network-probe have no rollback (classified accordingly).
• Audit trail: validation-audit.jsonl is append-only; the report embeds its sha256 so tampering is detectable.
• Expiry: targets.yaml#expires in the past aborts before Phase 1.
• Listener isolation: SSRF/callback probes bind a listener on 127.0.0.1 and reach it via kubectl port-forward reverse — no external egress required or permitted.

---

Output Layout

<analysis-results>/findings/<product>/<repo>/
├── <repo>-security-audit.{json,md}     # existing
├── <repo>-threat-model.md              # existing
├── <repo>-triage.{json,md}             # existing
├── <repo>-attack-plan.yaml             # new — reviewed plan
├── <repo>-validation.json              # new — schema-validated
├── <repo>-validation.md                # new — rendered
├── validation-audit.jsonl              # new — append-only action log
└── artifacts/                          # new — evidence files