validate-operator-live

name

validate-operator-live

description

Use when the user asks to live-validate, end-to-end test, or run the validation campaign for a logical product/operator from VALIDATION-PRIORITY-LIST.md against an ephemeral ROSA HCP cluster — provisioning the cluster via bonfire, installing the operator via the deploy-operator skill, generating a lab-environment targets.yaml, running the validate-findings skill with --auto --destructive, and releasing the cluster. One invocation == one cluster == one logical product.

Validate Operator Live (ROSA HCP)

End-to-end orchestration: provision → install → scope → validate → teardown for a single logical product on a single disposable ROSA HCP cluster. Composes the deploy-operator and validate-findings skills; itself composed by the 8-wide campaign loop in analysis-results/validations/_manifest/.

Authorization: Project Glasswing internal Red Hat assessment (Jira HCMSEC-3528). Each ROSA HCP cluster is provisioned exclusively for this run, declared environment: lab, and released on completion. Findings remain Red Hat internal / embargoed.

---

Input

/validate-operator-live <logical-product> [--keep-cluster] [--replay-only]
                        [--max-novel N] [--no-instance] [--duration 4h]

Flag	Effect
<logical-product>	Display name from progress-tracker/VALIDATION-PRIORITY-LIST.md (e.g. "ACM", "Compliance Operator"), or any alias product_map.py resolves.
--keep-cluster	Skip teardown (step 8). For interactive debugging only — the lease still expires.
--replay-only / --max-novel N	Passed through to validate-findings.
--no-instance	Do not apply the operand CR after operator install.
--duration	Initial bonfire lease (default 4h). One automatic 2h extension permitted (6h hard cap).

---

Environment

• KUBECONFIG=~/.kube/ephemeral.kubeconfig — must point at the bonfire hub (crc-eph). The ROSA cluster gets its own kubeconfig at /tmp/rosa-<ns>.kubeconfig; never confuse the two.
• bonfire pool list must show rosa.
• All paths below are relative to the workspace root (parent of ai-security-harness/).

---

Procedure

0. Resolve

python3 ai-security-harness/harnessing/deploy-operator/product_map.py "<logical-product>"

Capture: slug, olm_package, findings_packages[], instance_cr, namespace. If resolution falls through to the 1:1 default and the OLM package name looks wrong, stop and ask — a misnamed Subscription wastes a cluster.

Set output dir: OUT=analysis-results/validations/<slug> and mkdir -p $OUT.

1. Provision

bash ai-security-harness/harnessing/validate-operator-live/provision_rosa.sh <slug> --duration 4h \
  > "$OUT/cluster.json"

This runs bonfire deploy rosa --duration 4h (≈10–15 min), extracts the hub namespace, fetches the ROSA kubeconfig to /tmp/rosa-<ns>.kubeconfig, and writes {hub_namespace, kubeconfig, context, reserved_at, expires_at, extended:false} to $OUT/cluster.json.

If provisioning fails (no namespace in the log, or the kubeconfig secret is absent), record the failure (step 9) and do not proceed — but still attempt teardown if a namespace was reserved.

Read back: ROSA_KC=$(jq -r .kubeconfig $OUT/cluster.json), CTX=$(jq -r .context $OUT/cluster.json), HUB_NS=$(jq -r .hub_namespace $OUT/cluster.json).

2. Deploy operator + operand (agent-driven)

Follow the deploy-operator skill (ai-security-harness/harnessing/deploy-operator/SKILL.md) against the ROSA cluster (NOT the hub). This is a reasoning step — not a single script call:

Phase A (scripted):

python3 ai-security-harness/harnessing/deploy-operator/install_operator.py "<logical-product>" \
  --kubeconfig "$ROSA_KC" --context "$CTX" --timeout 1800 --out "$OUT"

If csv_phase != Succeeded: summarise $OUT/install-failure.yaml, record failure (step 9), teardown (step 8), stop.

Phase B (agent-driven operand setup): follow SKILL.md §B.1–B.5 — discover the primary CRD, read docs/alm-examples, create dummy Secrets/ConfigMaps via operand_setup.py dummy-secret, apply the operand CR, iterate on .status.conditions[] until Ready or budget spent, recording every dummy in $OUT/credentials-needed.json. Update $OUT/operator-install.json with operand_ready, operand_blocking, operand_kind/name, and the final credentials_needed[].

Capture from the result:

WATCHED=$(jq -r '.watched_namespaces|join(",")' $OUT/operator-install.json)
OPERAND_READY=$(jq -r '.operand_ready' $OUT/operator-install.json)

If operand_ready != true: the run is invalid-env — the attack surface is operator-only (CRD/RBAC/admission). You may still proceed to step 5 (those surfaces are worth probing), but the validation report will be flagged accordingly and any finding that needed the operand will be not_attempted: env.

3. Scope

python3 ai-security-harness/harnessing/validate-operator-live/gen_targets.py \
  --slug <slug> --context "$CTX" \
  --expires "$(jq -r .expires_at $OUT/cluster.json)" \
  $(for n in ${WATCHED//,/ }; do echo --watched-ns "$n"; done) \
  --out "$OUT/targets.yaml"

4. Lease check

bash ai-security-harness/harnessing/validate-operator-live/lease_guard.sh "$OUT/cluster.json"

Exit 3 means <30 min remain and the one extension is already spent — proceed with --replay-only regardless of user flags so something useful is recorded before expiry.

5. Validate

Invoke the validate-findings skill with the merged findings packages:

/validate-findings <pkg1>-findings,<pkg2>-findings,… \
  --targets $OUT/targets.yaml --context $CTX \
  --auto --destructive \
  [--replay-only] [--max-novel N] \
  --out $OUT

where <pkg1>,<pkg2>,… are the findings_packages from step 0. Set KUBECONFIG=$ROSA_KC for the duration of this step so the k8s adapter talks to the ROSA cluster.

This produces $OUT/<slug>-attack-plan.yaml, $OUT/<slug>-validation.json, $OUT/<slug>-validation.md, $OUT/validation-audit.jsonl, and $OUT/artifacts/.

6. Lease check (final)

Run lease_guard.sh again. If exit 3, skip any remaining novel-hunting and go straight to step 8.

7. Schema-validate

python3 ai-security-harness/scripts/validate_report.py \
  --schema ai-security-harness/schema/validation.schema.json \
  "$OUT/<slug>-validation.json"

8. Teardown (always)

Unless --keep-cluster:

bash ai-security-harness/harnessing/validate-operator-live/teardown_rosa.sh "$OUT/cluster.json"

This step runs even if any earlier step failed. Treat steps 1–7 as a try-block and step 8 as the finally-block.

9. Manifest update

python3 analysis-results/validations/_manifest/update_validation_progress.py \
  "<logical-product>" <status> \
  --hub-ns "$HUB_NS" \
  --confirmed N --refuted N --novel N \
  --report "$OUT/<slug>-validation.json"

Where <status> is done on success, failed otherwise. Counts come from $OUT/<slug>-validation.json#summary. Also append a one-line entry to analysis-results/validations/_manifest/PROGRESS.md.

---

Output Layout

analysis-results/validations/<slug>/
├── cluster.json               # hub_namespace, kubeconfig path, context, lease times
├── operator-install.json      # csv, operand_ready, owned_crds, credentials_needed
├── credentials-needed.json    # dummy creds applied + real creds required for pass-2
├── operand-setup.log          # transcript of agent-driven operand iteration
├── install-failure.yaml       # only on Phase-A failure
├── targets.yaml               # scope binding for validate-findings
├── <slug>-attack-plan.yaml
├── <slug>-validation.json     # schema-validated; includes needs_credential[] section
├── <slug>-validation.md
├── validation-audit.jsonl
└── artifacts/

Second-Pass Queue (real credentials)

Findings skipped with needs-credential:<ref> and operand prerequisites recorded with blocks_validation: true are aggregated across the campaign into analysis-results/validations/_manifest/credentials-needed.csv by the dashboard generator. That CSV is the worklist for a second validation pass where real credentials are supplied (via a --credentials <file> flag to validate-findings) for the subset of findings that genuinely need them.

---

Failure Modes & Recovery

Symptom	Likely cause	Action
bonfire deploy rosa times out at 1800s	ROSA pool exhausted or AWS quota	Record failed; do not retry within the same loop iteration (let the next worker try).
packagemanifest not found	OLM package name wrong in product_map.py	Fix the map; mark failed with reason.
CSV stuck Pending / InstallReady	OperatorGroup conflict or missing dependency	Inspect install-failure.yaml; some operators (ODF, ACM) need pre-req operators — add to instances/ as a multi-doc YAML.
validate-findings blocked_by_scope for every step	KUBECONFIG still points at hub	Confirm $ROSA_KC exported for step 5.
Lease expired mid-run	Heavy operator + large findings set	Result counts up to that point are still in validation-audit.jsonl; report partial; consider re-running with --replay-only on a fresh cluster.