deploy-operator

name

deploy-operator

description

Use when the user asks to install, deploy, or stand up an OLM operator (or a logical product from VALIDATION-PRIORITY-LIST.md) onto an OpenShift/Kubernetes cluster AND bring its operand to a Ready state — resolving the OLM package, creating the Subscription, waiting for CSV Succeeded, then reading the operator's documentation/samples to discover operand prerequisites, creating dummy Secrets/ConfigMaps to satisfy them, applying the operand CR, iterating on its status conditions until Ready, and recording every dummy credential in credentials-needed.json for a later real-credential pass.

Deploy Operator — with Agent-Driven Operand Setup

Install an OLM operator AND bring a representative operand to Ready so the validate-findings harness has a live attack surface. The operator install is scripted; the operand setup is reasoning-driven: you read the operator's documentation, discover what the operand CR needs, create dummy credentials/config to satisfy it, and iterate on status conditions until it's running — recording every dummy you applied so a second validation pass can substitute real values.

Why agent-driven? Each operator's operand has different prerequisites (S3 buckets, GitHub tokens, TLS certs, dependency operators, ingress hostnames). Rather than hand-write a kit per operator, the agent discovers requirements at runtime from the CSV's alm-examples, the CRD schema, the operator's README, and the operand's own .status.conditions[] error messages.

---

Input

/deploy-operator <target> [--kubeconfig <path>] [--context <ctx>]
                 [--channel <ch>] [--namespace <ns>] [--source <catalog>]
                 [--timeout <secs>] [--out <dir>]
                 [--no-operand]            # CSV only; skip operand setup
                 [--operand-budget <N>]    # max fix-iterate cycles (default 4)

<target> is a logical-product display name (e.g. "ODF", "3scale / APIcast"), a findings-package slug, or a raw OLM package name — resolved via harnessing/deploy-operator/product_map.py.

--out <dir> (typically analysis-results/validations/<slug>/) receives operator-install.json, credentials-needed.json, and operand-setup.log.

---

Phase A — Operator Install (scripted)

python3 harnessing/deploy-operator/install_operator.py "$TARGET" \
  --kubeconfig "$KC" --context "$CTX" --timeout 1800 --out "$OUT"

Applies Namespace + OperatorGroup + Subscription, polls CSV → Succeeded, writes $OUT/operator-install.json. Capture from it: csv, namespace, owned_crds[], watched_namespaces[].

If csv_phase != Succeeded: dump diagnostics, stop — operand setup is pointless without an operator.

If --no-operand: report and stop here.

---

Phase B — Operand Setup (agent-driven)

Goal: at least one instance of the operator's primary owned CRD reaches Ready/Available. Budget: --operand-budget fix-iterate cycles (default 4) or 20 minutes, whichever first.

All oc commands below use --kubeconfig "$KC" --context "$CTX". Log every action to $OUT/operand-setup.log.

B.1 — Discover the operand kind and a starting CR

1. Pick the primary CRD from owned_crds[]: usually the one whose kind matches the product (e.g. APIManager, QuayRegistry, HyperConverged, MultiClusterHub). If unclear, prefer the one with the longest description or the first in the list.

2. Get a starting CR, in priority order:

   • harnessing/deploy-operator/instances/<olm_package>.yaml (or instance_cr from product_map) if present — already has {{NAMESPACE}}/{{INGRESS_DOMAIN}} placeholders.
   • python3 harnessing/deploy-operator/operand_setup.py sample-cr <crd-name> --kubeconfig $KC — the CSV's alm-examples annotation.
   • The CRD schema's required fields with minimal values.

3. Resolve placeholders:

   DOMAIN=$(python3 harnessing/deploy-operator/operand_setup.py ingress-domain --kubeconfig $KC)
   

   Substitute {{NAMESPACE}} → install namespace, {{INGRESS_DOMAIN}} → $DOMAIN in the CR.

B.2 — Read documentation for prerequisites

All operators in this campaign ship from the redhat-operators catalog, so each has official Red Hat product documentation that describes the customer-facing install and configuration procedure — including prerequisite Secrets, ConfigMaps, dependency operators, and sample operand CRs. Use these as the primary source: an operand configured per the official docs is what we'd expect to find in a customer cluster, so it's the right shape to validate against.

Resolve doc sources for the installed version:

python3 harnessing/deploy-operator/operand_setup.py doc-sources \
  "$OLM_PACKAGE" --csv "$CSV" --kubeconfig "$KC"

This returns a prioritised JSON blob; consult in this order (≤ ~6 fetch calls total — the iterate loop in B.4 catches anything missed):

1. Catalog entry (packagemanifest.links[], packagemanifest.long_description_excerpt, csv_annotations) — the OperatorHub tile content. Often links directly to "Installing " and lists operatorframework.io/suggested-namespace. Already in the JSON; no fetch needed.

2. CSV spec.description (csv_description) — many operators embed the full prerequisites section here (Asciidoc/Markdown). Already in the JSON; no fetch needed.

3. docs.redhat.com versioned product docs (rh_docs[0]) — https://docs.redhat.com/en/documentation/<product>/<X.Y>. Fetch the index, then the "Installing" and "Configuring " chapters (look for headings matching the operand Kind). The version <X.Y> is taken from the CSV so you read the doc revision that matches what you installed. If the versioned URL 404s, fall back to rh_docs[1] (unversioned latest). This host is reachable on the Red Hat VPN with Kerberos/SSO — use WebFetch or curl --negotiate -u : for authenticated pages.

4. access.redhat.com knowledgebase (rh_search) — solution articles for known prerequisite gotchas (" operand stuck Pending", "secret X not found"). Fetch only if step 3 didn't answer a specific blocker from B.4.

5. CRD OpenAPI schema — oc get crd <name> -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema}' for the exact required fields and any secretRef / configMapRef / *Credentials field names the docs assume you've created.

6. Upstream repo README (upstream[]) — last resort when the productised docs lag a CRD field; treat upstream-only config as lower-confidence (a customer wouldn't have read it).

From these, build a prerequisite list: Secrets the operand expects (name, keys, what they authenticate to), ConfigMaps, dependency operators (with their own Subscriptions), StorageClass, hostnames. Note in operand-setup.log which doc URL each prerequisite came from — that's the citation if a customer asks "why did you configure it this way".

B.3 — Create prerequisites (real if available, else dummy)

For every Secret/ConfigMap the operand references:

First, if GLASSWING_REAL_CREDS=1 is set in the environment, try the real-credential resolver:

python3 harnessing/deploy-operator/real_creds.py resolve <cred-name> \
  -n $NS --secret-name <secret-name> --product-slug $SLUG \
  --kubeconfig $KC --out $OUT --apply

Exit codes: 0 = real Secret applied (and recorded in credentials-needed.json with real_applied:true) — skip the dummy step for this prerequisite. 2 = no resolver / real-creds mode off — fall through to dummy below. 3 = resolver exists but no real value available (e.g. llm-provider-apitoken) — fall through to dummy and keep blocks_validation:true. The resolver may provision AWS resources on demand (shared S3 bucket + IAM user, per-cluster IAM role trusting the ROSA OIDC issuer, Route53 zone); all are tagged glasswing-ephemeral=true for later sweep. Use real_creds.py resolve <name> --json (no --apply) when you need the raw values (AWS_S3_BUCKET, ROLE_ARN, ZONE_ID) to wire into the operand CR spec rather than a Secret.

Otherwise (or on fall-through), create a dummy:

python3 harnessing/deploy-operator/operand_setup.py dummy-secret \
  s3-credentials -n $NS \
  --key AWS_ACCESS_KEY_ID=aws_access_key_id \
  --key AWS_SECRET_ACCESS_KEY=aws_secret_access_key \
  --key BUCKET_HOST=endpoint --key BUCKET_NAME=bucket \
  --kubeconfig $KC --out $OUT

The =hint suffix selects a plausible dummy shape (so regex/length validators pass). Every call appends to $OUT/credentials-needed.json. Never invent credentials. All dummies start with glasswing-dummy so they're greppable.

For dependency operators (e.g. SMCP needs Kiali + Jaeger; ODF needs local-storage), create additional Subscriptions in the same namespace and wait for their CSVs.

B.4 — Apply, observe, fix, iterate

oc apply -f /tmp/operand.yaml

Then loop up to --operand-budget times:

python3 harnessing/deploy-operator/operand_setup.py operand-status \
  <Kind> <name> -n $NS --kubeconfig $KC

Returns {ready, conditions[], blocking}. If ready: true → done. Otherwise read blocking and act:

Blocking message pattern	Fix
secret "<name>" not found / *SecretRef missing	dummy-secret <name> -n $NS --key <inferred> --out $OUT
configmap "<name>" not found	dummy-configmap <name> -n $NS --kv key=glasswing-dummy --out $OUT
no matches for kind "<K>" / CRD missing	Install the dependency operator (Subscription)
storageclass .* not found	Set .spec.*.storageClassName to the cluster default (oc get sc -o name) or remove the field
hostname / domain invalid	Replace with <name>.$DOMAIN
auth / 401 / 403 / unable to authenticate	The dummy credential is rejected by an external service. Record it as blocks_validation: true and either (a) stub the external dependency (point URL at a harness-controlled listener) or (b) accept the operand will be Degraded and proceed — many findings target the controller, not the external call.
insufficient .* resources / Pending scheduling	Reduce replicas/resources in the CR; on ROSA HCP add nodeSelector: worker if the operand pins to master.
External databases are required / embedded * is not supported	Operator version dropped embedded backing services (e.g. 3scale ≥2.16 needs external Redis/Postgres). Deploy lightweight in-cluster instances (registry.redhat.io/rhel9/redis-7, rhel9/postgresql-15) + the connection Secrets the docs name, then set spec.externalComponents.*=true (or equivalent) on the operand CR.
token audiences ... invalid / istiod XDS auth fails / JWT_POLICY=first-party-jwt	ROSA HCP (and any HyperShift guest) uses bound SA tokens with an OIDC issuer; maistra/istio auto-detects first-party-jwt and fails. Patch the operand with spec.security.identity.type: ThirdParty (SMCP) or the equivalent JWT_POLICY=third-party-jwt setting.
Anything you can't fix in one step	Record it: operand_setup.py record-cred --out $OUT --name <what> --kind external --reason "<message>" --blocks-validation

After each fix, oc apply the updated CR (or just wait if the fix was a prerequisite object) and re-poll. Sleep 20–30 s between polls.

B.5 — Decide and record

After the budget is spent or ready: true:

• Set operand_ready in $OUT/operator-install.json (re-read, mutate, rewrite the file).
• If not ready: set operand_ready: false, operand_blocking: <last blocking message>, and ensure the blocker is in credentials-needed.json with blocks_validation: true.
• Append the operand's namespace and any namespaces it created to watched_namespaces[].

Do not treat operand-not-ready as a failure of this skill — the operator is up, and many findings target operator-level surfaces (admission webhooks, RBAC, CRD validation) that are testable without the operand. The downstream validate-operator-live skill decides whether to proceed based on operand_ready.

---

Phase C — Credential-Gated Validation Steps

Some findings can only be meaningfully validated with a real credential (e.g. "SSRF to GitHub API leaks repo contents" needs a real token to observe the leak). When you recognise such a finding during operand setup or from the prerequisite list, record it with --blocks-validation so the validate-findings harness can:

1. Mark the corresponding step not_attempted with reason needs-credential:<name> instead of running it with a dummy and getting a meaningless inconclusive.
2. List it in the validation report's credentials_needed[] section.
3. Queue it for the second-pass run where real credentials are supplied via --credentials <file>.

The validate-findings harness reads $OUT/credentials-needed.json and cross-references finding PoCs that mention any name/key listed with blocks_validation: true.

---

Output

$OUT/operator-install.json (extended):

{
  "logical_product": "...", "olm_package": "...", "csv": "...",
  "csv_phase": "Succeeded",
  "namespace": "...", "watched_namespaces": [...],
  "owned_crds": [{"kind":"...","name":"...","version":"..."}],
  "operand_applied": true,
  "operand_ready": true|false,
  "operand_blocking": "<last status-condition message if not ready>",
  "operand_kind": "...", "operand_name": "...",
  "credentials_needed": [ ... mirror of credentials-needed.json ... ],
  "elapsed_s": ..., "status": "ok"
}

$OUT/credentials-needed.json:

[
  {"name":"s3-credentials","namespace":"openshift-storage","kind":"Secret",
   "keys":["AWS_ACCESS_KEY_ID","AWS_SECRET_ACCESS_KEY","BUCKET_HOST","BUCKET_NAME"],
   "reason":"operand prerequisite — dummy value applied",
   "dummy_applied":true,"blocks_validation":false},
  {"name":"github-token","namespace":"","kind":"external",
   "keys":["GITHUB_TOKEN"],
   "reason":"PoC for FIND-042 calls api.github.com; dummy gives 401",
   "used_by":"FIND-042","dummy_applied":false,"blocks_validation":true}
]

$OUT/operand-setup.log: free-form transcript of B.1–B.5 (what you read, what you created, each iterate cycle's blocking message and fix).

---

Failure modes

Symptom	Action
CSV never Succeeded	Phase A failure — dump install-failure.yaml, stop.
Operand CRD admission rejects every variant	Record the rejection in credentials-needed.json (kind external, blocks_validation: true), set operand_ready: false, proceed.
Dependency operator's CSV stuck	Treat as Phase-A-style failure for the dependency; record and proceed without it.
Docs unreachable (VPN/network)	Skip B.2; rely on alm-examples + iterate loop.