| name | cluster-install |
| description | Use when installing Cozystack on an existing Kubernetes cluster (kubeadm / k3s / RKE2 / managed). Discovers cluster facts, validates node readiness via kubectl debug, recommends an installer + platform variant, gathers values interactively, creates the ZFS pool on each storage node through kubectl debug (cozystack standardises on ZFS for LINSTOR — LVM / LVM-thin paths are not supported), installs extractedprism (per-node kube-apiserver HA proxy, generic variant default), installs the cozy-installer chart, applies the Platform Package, patches the root Tenant for ingress (breaks the OIDC chicken-and-egg), waits until every HelmRelease is Ready, prints a NOTES-style access summary, and offers an issue-template handoff to cozystack/* on fatal failure. Not for Kubernetes bootstrap and not for upgrades — see `cozystack:cluster-upgrade` for that. Talos node-prep is out of scope — `cozystack:cluster-install` refuses if Talos nodes are missing cozystack-tuned extensions and points at `cozystack:talos-bootstrap`. |
| argument-hint | [--config-dir=<path>] [--context=<kube-context>] [--installer-version=<vX.Y.Z>] [--installer-variant=<talos|generic|hosted>] [--platform-variant=<isp-full|isp-full-generic|isp-hosted|default>] [--no-extractedprism] [--api-host=<ip>] [--dry-run] |
cozystack:cluster-install
Work in reasoning mode. Use the phrasing cozystack:cluster-install (not "the skill") in user-facing messages. Announce phase transitions: cozystack:cluster-install Phase N — <name>.
Note on language in this SKILL.md — every operator-facing prompt below is written in English for clarity. At runtime the skill matches the operator's natural language detected from prior conversation messages (or read from <config-dir>/.state.yaml operator_language when the wizard chain is in progress). Code identifiers, commands, file paths, and any text destined for GitHub stay canonical.
Entry point note: cozystack:wizard is the recommended way to reach this skill. The wizard runs node-bootstrap (talos-bootstrap / ubuntu-bootstrap) before handing off here when needed. This skill is also callable directly when the operator already has a ready cluster with prepared nodes.
Source of truth, in priority order:
- Live cluster state —
kubectl --context $CTX ....
- Upstream chart values:
~/git/github.com/cozystack/cozystack/packages/core/{installer,platform}/values.yaml and the variant overlays.
- Install guide for the major matching
--installer-version: https://cozystack.io/docs/v<X.Y>/install/kubernetes/generic/ (or talos/, air-gapped, etc.).
- Ansible reference:
~/git/github.com/cozystack/ansible-cozystack/roles/cozystack/{defaults,tasks}/main.yml.
Never guess versions, IPs, label values, or CIDRs — read them from the cluster or ask.
Core principles
- Match the operator's natural language. Read from
<config-dir>/.state.yaml operator_language (set by cozystack:wizard Phase 0) or detect from prior messages when invoked directly. Use it in prompts, AskUserQuestion options, summaries, and gates. Code identifiers, commands, file paths, and GitHub-public text stay in their canonical form.
- One valid path → just do it. After the operator approved the consolidated plan in Phase 5 (STOP GATE 2), the skill runs helm install + Platform Package apply + Tenant patch + HR wait + verification back-to-back without re-prompting. Approval gates remain only for (a) multi-option questions in Phase 4 (storage layout, network values, publishing host), (b) destructive operations (Phase 5.5 per-node storage provisioning — each
zpool create is a real choice with data implications), (c) STOP GATEs 1/2/3 themselves. No "are you ready to continue?" between phases that have one valid outcome.
- Front-load the interview. Every question the skill might ask in any phase is collected upfront in Phase 4. That includes per-node storage devices (Phase 5.5), per-node provisioning approvals (Phase 5.5), extractedprism opt-out (Phase 5.6), Tenant ingress patch confirmation (Phase 8, inline), and HR stuck-state recovery preferences (Phase 8). Phase 2 cluster lookup + Phase 0
intent_hints from <config-dir>/.state.yaml must fill every slot they can before any question fires. Phase 4 presents one consolidated summary with every slot filled (defaults marked) and quick-edit affordances. Phases 5.5 onward execute against the collected answers — no re-prompts mid-flow except destructive STOP GATEs that have to ask by their nature (e.g. zpool destroy on an existing pool the operator chose to wipe).
- Layer-pure operator output. The skill never says "returning control to wizard", "the wizard will dispatch next", or any other orchestration commentary in the operator-facing summary. Whoever invoked the skill (a human running
/cozystack:cluster-install directly, or the wizard's dispatch loop) figures out what's next on their own. Internal SKILL.md references to cozystack:wizard are fine for documentation; wizard does not appear in any text shown to the operator.
- The user is click-ops. Show what you're about to do, in plain language, before doing it. Wait for
Continue.
- Three non-negotiable gates: cluster fits, values gathered, all HRs Ready. None can be skipped.
- Read-only lookups (
get, describe, logs, ephemeral debug pods) need no approval. Mutating actions (apply, patch, helm install, label) need explicit user approval each time.
- Errors are not the user's fault. When a check fails, explain what's wrong, why it matters, and offer concrete next steps (with commands). Don't leave the user staring at a Helm stack trace.
- On a fatal failure that looks upstream: stop, assemble a diagnostic bundle (
references/issue-templates.md), draft an issue body, hand it to the user.
Phase 1 — Parse arguments and pin the context
Parse flags. Defaults:
--context → use the one you parse from kubectl config current-context but show it to the user and ask Use this context or pick another? via AskUserQuestion. Do not silently inherit current-context.--installer-version → if absent, ask. Suggest the latest known stable from the cozystack repo's tags (read git -C ~/git/github.com/cozystack/cozystack tag --list 'v*' --sort=-v:refname | head -5).--installer-variant and --platform-variant → recommended by Phase 2 unless preset.
After the user picks a context, run and show:
kubectl --context $CTX cluster-info | head -1
kubectl --context $CTX get nodes --output wide
Detect prod-like signals (prod, production, prd in the context name). If present, refuse without --allow-prod and explain. The user must re-invoke with the flag.
Check for an existing install:
kubectl --context $CTX get namespace cozy-system --ignore-not-found --output name
kubectl --context $CTX get package cozystack.cozystack-platform --ignore-not-found --output name
If cozy-system exists and holds resources (any pods, any HRs, any operator deployment), refuse — point the user at cozystack:cluster-upgrade. The only "exists but empty" case to handle is a stale namespace with no helm metadata (see Phase 6 namespace-adoption).
Phase 2 — Cluster lookup and variant recommendation
All read-only. Don't ask the user anything yet — gather facts first, then present.
Collect:
- k8s version (
kubectl version --output json).
- Per-node
nodeInfo.osImage, kernelVersion, kubeletVersion, architecture, kubeProxyVersion, plus all labels and status.addresses.
- Cluster domain — from coredns Corefile (see
references/node-checks.md). Must be cozy.local.
- Pod CIDR —
kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'.
- Service CIDR — try
kubectl --namespace kube-system get pod --selector component=kube-apiserver -o yaml; fall back to kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}' and infer.
- CNI pods in
kube-system.
- Conflicting workloads (ingress-nginx, cert-manager, metrics-server, kube-proxy, traefik, servicelb).
- Existing storage classes and the default.
- Existing LoadBalancer-class services.
- Control-plane node labels:
node-role.kubernetes.io/control-plane key and value per node (see references/node-checks.md).
- Storage discovery (per node, via
kubectl debug node in read-only mode — lsblk, pvs, vgs, lvs --all, zpool list, command -v pvcreate vgcreate lvcreate lvs; on Talos additionally lsmod for drbd/zfs/openvswitch and /etc/lvm/lvm.conf global_filter). Records: unmounted devices ≥ 50 GiB, existing VGs / LVs / zpools (will be reused as Phase 4 defaults), LVM-tools availability, Talos extension readiness. Skip on isp-hosted.
Apply variant recommendation logic from references/variants.md. Print:
cluster lookup
context: $CTX
api server: $API_URL
distribution: $DIST (kubelet $K8S_VERSION)
nodes: $N (cp: $CPN, worker: $WN, mixed-arch: $MIXED)
cluster domain: $DOMAIN (required: cozy.local — $MATCH)
pod CIDR: $POD_CIDR
service CIDR: $SVC_CIDR (apiserver flag: $SVC_FLAG_MATCH)
cni: $CNI_PODS ($CNI_VERDICT)
conflicts: $CONFLICT_LIST or "none"
cp label values: $CP_LABEL_VALUES (per node)
storage:
per-node disks: $UNMOUNTED_DISKS_PER_NODE
existing pools: $EXISTING_VGS_OR_ZPOOLS or "none"
lvm tools: $LVM_TOOLS_VERDICT
talos modules: $TALOS_DRBD_ZFS_OVS_VERDICT (talos only)
recommended:
installer variant: $INSTALLER_VARIANT
platform variant: $PLATFORM_VARIANT
why: $REASON
Hard refusals (don't move on, surface clearly):
- Cluster domain is not
cozy.local → refuse, link to bootstrap docs for the user's distribution.
- CNI conflict (non-hosted) → refuse, list pods to remove.
- Conflicting workloads (ingress-nginx, cert-manager, etc., non-hosted) → refuse, list workloads.
- kubectl can't reach the cluster → refuse, suggest
kubectl auth can-i '*' '*' --all-namespaces.
Phase 3 — Node readiness validation
Skip on isp-hosted (no node access required).
For every other case, drive kubectl debug node per references/node-checks.md. For homogeneous clusters, check one sample CP node and one sample worker; warn that the rest are assumed identical. For heterogeneous, check every node.
Talos-specific early-exit. If Phase 2 detected Talos on any node, run the four Talos checks from references/node-checks.md (lsmod drbd / lsmod zfs / lsmod openvswitch / /etc/lvm/lvm.conf cozystack global_filter). If any of them fail on any Talos node, STOP GATE 1 fails immediately with this message:
Talos nodes detected without cozystack-tuned extensions.
Missing on $NODE: $MISSING_LIST (drbd / zfs / openvswitch / lvm.conf filter)
`cozystack:cluster-install` installs Cozystack on top of an already-prepared
cluster — Talos node prep is out of scope.
Fix paths:
- Run `/cozystack:talos-bootstrap` (separate skill; prepares Talos nodes).
- Or reinstall the affected nodes from the cozystack-tuned image
`ghcr.io/cozystack/cozystack/talos:vX.Y.Z` (see
https://cozystack.io/docs/v1.3/install/talos/).
Refusing to proceed — no mutations performed.
No partial install on a half-tuned Talos cluster.
For non-Talos and tuned-Talos, aggregate findings into a node readiness matrix:
node-readiness
node1 (cp): kmod ✓ pkgs ✓ svcs ✓ sysctl ✓ multipath-bl ✓ disks ✓ cp-label ✓ storage-tools ✓
node2 (cp): kmod ✓ pkgs ✓ svcs ✗ (iscsid inactive) sysctl ✓ multipath-bl ✗ ...
node3 (worker): ...
For every ✗ surface: the exact check that failed, the value observed vs required, and one concrete fix command. If many nodes share the same gap on Ubuntu / Debian, the recommended fix is to run cozystack:ubuntu-bootstrap first (it wraps cozystack/ansible-cozystack/examples/ubuntu/prepare-ubuntu.yml which covers every node-prep concern Cozystack has). On Talos, the fix is cozystack:talos-bootstrap. For other distributions (RHEL family, SUSE), point operators at cozystack/ansible-cozystack/examples/{rhel,suse}/prepare-*.yml directly — out of scope for v1 of cozystack:wizard.
STOP GATE 1 — Cluster readiness
Present:
gate 1 — cluster readiness
summary: <pass | needs work | refused>
blockers:
- <list of refusal reasons; empty if "pass">
warnings:
- <list of soft issues; e.g. mixed arch nodes, single-node cluster, weak resources>
next:
options: Continue / Fix and re-check / Cancel
If blockers present → only Cancel is honoured. If only warnings → user picks.
Phase 4 — Consolidated intake
This is the one interview phase. Policy slots are read from state.cozystack_intake (written by cozystack:wizard Phase 4 — the chain orchestrator front-loads every operator-decidable value before any skill runs). Discovery slots — the ones that need post-bootstrap probing (actual device paths, KubeOVN label values, real Node.status.addresses) — are resolved here against the live cluster.
Two-tier read pattern:
- Read
state.cozystack_intake first. Every value the wizard collected is the operator's authoritative answer; this skill never re-prompts what's already there.
- Discovery-driven fill. Run Phase 2's lookups and Phase 3's node checks against the policy values. Pre-fill remaining slots from the discovery output: largest unmounted disk per node (matching
cozystack_intake.storage_pref.layout_per_node[<name>]), real InternalIP / ExternalIP (validates cozystack_intake.external_ips.strategy), KubeOVN MASTER_NODES label state, talos KubePrism presence.
- Render the consolidated summary with every slot filled and a single
Approve all / Edit <slot> / Cancel gate. AskUserQuestion fires only when (a) cozystack_intake is missing — direct invocation without the wizard, (b) discovery contradicts the policy (e.g. external_ips.strategy: external but every node's ExternalIP == InternalIP), or (c) destructive STOP GATEs that have to ask by their nature.
When state.cozystack_intake is absent (operator ran /cozystack:cluster-install directly without the wizard), the skill falls back to asking each slot inline — same shape as before the front-load refactor. This keeps direct invocation viable; running through the wizard is the optimised path.
The shape of the consolidated summary (operator sees this once, not 10 times):
cozystack:cluster-install — collected values
bundles: system, paas, iaas, naas (default for platform_variant: isp-full-generic on installer_variant: generic)
storage (ZFS):
cp1 (10.0.0.10): /dev/nvme1n1 → zpool data (single)
cp2 (10.0.0.11): /dev/nvme1n1 → zpool data (single)
cp3 (10.0.0.12): /dev/nvme1n1 → zpool data (single)
linstor pool name: data
networking:
podCIDR: 10.244.0.0/16 (cozystack default; matches apiserver: ✓)
podGateway: 10.244.0.1
serviceCIDR: 10.96.0.0/16 (cozystack default; matches apiserver: ✓)
joinCIDR: 100.64.0.0/16
apiServerHost: 127.0.0.1 (extractedprism, default)
apiServerPort: 7445
kubeovn MASTER_NODES: "" (Helm lookup, label matches)
publishing:
external IPs: 10.0.0.50 (from MetalLB pool / operator-supplied)
mode: externalIPs
host: 10-0-0-50.nip.io (nip.io default; ownership gate auto-passes)
apiServerEndpoint: https://api.10-0-0-50.nip.io
exposed: api, dashboard
cert solver: http01
operations:
storage provisioning: auto (one approve per node in Phase 5.5)
extractedprism: enabled (default for generic)
tenant ingress patch: enabled (default for system bundle)
options:
- Approve all — proceed to Phase 5 plan gate
- Edit <slot> — name the slot (storage, networking.podCIDR, publishing.host, …)
- Cancel
Slot legend (every slot the operator may want to edit):
-
Bundles (multiSelect, defaults from variant overlay):
- system (required for
isp-full*; off for isp-hosted)
- paas (databases / applications)
- iaas (Cluster API + VMs)
- naas (Network as a Service)
-
Storage (only when system bundle is on — isp-hosted skips). Cozystack standardises on ZFS for LINSTOR pools; see references/storage-backends.md. No backend question — only device selection and pool layout per node.
a. Per-node disk selection — for every storage-providing node, show Phase 2's unmounted-device list and ask which device(s) to use. Default is the largest unmounted disk ≥ 50 GiB. If a node has multiple candidates or the operator wants a mirror / RAID-Z, prompt for a vdev layout:
single — one disk, no redundancy (Recommended for dev / sandbox).mirror — two disks, two-way mirror (Recommended for prod on 2-disk nodes).raidz — 3+ disks, one parity disk.
Refuse and ask again if the node has no qualifying device — operator picks: re-check / cancel / proceed without this node (the latter excludes the node from storage).
b. Names:
- zpool name — default
data. Same name on every storage node (the LINSTOR storage-pool entry surfaces them under one logical name).
- LINSTOR storage-pool name — default
data. What linstor storage-pool list shows; referenced by every StorageClass parameters.linstor.csi.linbit.com/storagePool.
c. Summary echo after collection:
storage decision (ZFS)
node1: /dev/nvme1n1 → zpool data (single)
node2: /dev/nvme1n1, /dev/nvme2n1 → zpool data (mirror)
node3: /dev/nvme1n1 → zpool data (single)
linstor pool: data
Refuse to proceed if any storage node lacks ZFS tooling (zpool / zfs binaries) — the Phase 3 storage discovery would have caught it, but re-verify here in case a fix was applied since. RHEL 10 family is not supported on the storage path; see references/known-failures.md.
-
podCIDR / serviceCIDR / joinCIDR — show detected as defaults. If user picks Other, validate format (CIDR notation, no overlap with host networks the cluster sees, joinCIDR ≠ podCIDR ≠ serviceCIDR).
-
podGateway — auto-derive as the first IP of podCIDR, confirm.
-
apiServerHost — the address Cilium / KubeOVN / cozystack-operator dial to reach kube-apiserver. The skill picks this automatically by installer variant — do not ask the operator unless they passed --api-host=<ip>.
talos variant — localhost:7445 (KubePrism, built into Talos machine-config). Operator can't override; Cozystack's values-isp-full.yaml overlay hard-codes this for Cilium.generic variant — default — 127.0.0.1:7445 via extractedprism DaemonSet (a per-node TCP load balancer for kube-apiserver HA; mirrors what KubePrism does on Talos). The skill installs the chart in Phase 5.6 before the cozy-installer chart. No VIP, no keepalived. Single-CP sandboxes work too — extractedprism just proxies to one endpoint.generic variant with --no-extractedprism — operator must supply --api-host=<ip> (internal IP of a CP, or a VIP / external LB IP they manage themselves). Single point of failure if a single CP IP is given on a multi-CP cluster; the skill warns but does not refuse.hosted variant — the managed provider handles kube-apiserver HA; the skill does not install extractedprism and does not set apiServerHost (the cozy-installer chart's hosted variant doesn't need it).
The plan presentation in Phase 5 always shows which choice landed and how to flip it.
-
LB / external IPs —
-
Mode: externalIPs (recommended for now; deprecated upstream in k8s v1.36) vs loadBalancer (Cilium L2/BGP or external cloud LB; needs more wiring).
-
Pool: resolved from state.cozystack_intake.external_ips (wizard Phase 4). The wizard already asked the operator's strategy — internal / external / explicit — and the reason. Here the skill validates the strategy against live Node.status.addresses and refuses to silently override:
kubectl --context $CTX get nodes --output json \
| jq -r '.items[] | {name: .metadata.name,
internal: (.status.addresses[] | select(.type=="InternalIP") | .address),
external: (.status.addresses[]? | select(.type=="ExternalIP") | .address // "")}'
Cases:
strategy: internal and every node's InternalIP is present → use those.strategy: external and every node's ExternalIP is present and differs from InternalIP → use the external set, but first print the NAT-warning probe: surface the strategy + reason from intake, and warn that on OCI 1:1 NAT / GCP NAT'd external IPs / AWS EIP the public IP is not present on the interface that receives the packet — the kernel only sees the InternalIP. Picking external here on such providers causes Cilium externalIPs BPF to never match, producing Connection refused on the dashboard host even when every HR is Ready. If intent_hints.platform ∈ {oci, aws-with-eip, gcp-with-nat}, refuse the external choice and force internal with a one-line justification; the operator can override with --allow-external-on-nat-provider if they know better.strategy: explicit → take cozystack_intake.external_ips.explicit verbatim; sanity-check that each IP is reachable from at least one node (nc -zw1 <ip> 80 from the first CP via kubectl debug if --check-externals is set; otherwise informational only).cozystack_intake.external_ips is missing (direct invocation without the wizard) → ask inline. Default internal when InternalIP ≠ ExternalIP on any node. Surface the NAT warning.
The chosen pool is recorded as cozystack.publishing.external_ips and lands in the Platform Package CR spec.components.platform.values.publishing.externalIPs.
Collect these before the publishing.host question — the domain gate references the chosen IPs.
-
publishing.host (FQDN) + domain ownership gate — this is the public domain under which every service lives: dashboard.${HOST}, keycloak.${HOST}, api.${HOST}, grafana.${HOST}. Cozystack creates a wildcard ingress for *.${HOST} and asks Let's Encrypt for certificates via the HTTP-01 solver by default (publishing.certificates.solver: http01).
This means the domain must be:
- Owned by the operator — they need to configure DNS for it.
- Publicly resolvable — wildcard
*.${HOST} A-records pointing at the external IPs picked in question #6.
- Reachable on port 80 from the public internet — Let's Encrypt validators hit
http://<challenge-host>/.well-known/acme-challenge/....
Failure modes if any of these is wrong: cert-manager Order CRs go pending, certificates never issue, ingress serves a default cert that browsers reject, dashboard / keycloak unreachable.
Options:
a) Custom FQDN (cluster.example.com) — Recommended for real deployments. Hard gate: show this confirmation:
You picked publishing.host = cluster.example.com
Cozystack will request Let's Encrypt certificates for *.cluster.example.com
via the HTTP-01 solver. For this to work, you must:
1. Own the domain cluster.example.com (or its parent example.com).
2. Configure a wildcard A-record: *.cluster.example.com → <EXTERNAL_IPS>
(the same IPs you picked at the "external IPs" question above).
3. Make port 80 reachable from the public internet at those IPs.
If any of this is not true, certificates will never issue, every browser
visit to https://dashboard.cluster.example.com will warn, and the install
verification at Phase 9 will fail the dashboard reachability probe.
Confirm: I own this domain and DNS will be configured before this install completes.
options: Yes, I own this domain / No, let me pick a different host / Cancel install
Without an explicit yes — do not proceed.
b) nip.io playground (<LB-IP-with-dashes>.nip.io, e.g. 192-0-2-10.nip.io) — Recommended for sandbox / dev. Works out of the box: nip.io is a public service that resolves <IP-with-dashes>.nip.io to the embedded IP, Let's Encrypt accepts nip.io domains, no DNS configuration needed. Skip the ownership gate for nip.io patterns.
After collection, run a soft DNS pre-flight (warning only, not a refusal — operator may be configuring DNS in parallel):
probe="precheck-$(TZ=UTC date +%s).${HOST}"
dig +short "$probe" | head -3
On a nip.io host this returns the embedded IP immediately. On a fresh custom FQDN it may return empty until DNS propagates. Surface either way:
DNS pre-flight for *.${HOST}: $RESULT
Expected: at least one of <EXTERNAL_IPS> appears in the output.
Got: $DIG_OUTPUT
- If matches: DNS already configured. Proceed.
- If empty / different: configure DNS now (wildcard A-record) before
Phase 8 reaches dashboard / keycloak HRs (~10 min into wait).
The install will still progress, but will fail those HRs until DNS
resolves.
options: Continue (DNS will be configured shortly) / Re-check DNS / Cancel
-
Cert-manager solver — http01 (default, Recommended — only works with public DNS + port 80) vs dns01 (works on internal networks, but needs DNS-provider credentials in cert-manager values, out of scope for v1 of cozystack:cluster-install). On nip.io always pick http01. If operator insists on dns01, set publishing.certificates.solver: dns01 and remind them the issuer config has to be applied manually.
-
apiServerEndpoint — default https://api.<publishing.host>:6443. Explain it goes into client kubeconfigs.
-
exposedServices (multiSelect): api, dashboard, vm-exportproxy, cdi-uploadproxy. Default api,dashboard.
-
KubeOVN MASTER_NODES — branch on Phase 2 finding:
- If CP label value matches variant expectation on at least one node → default empty (let lookup work).
- Otherwise → pre-fill comma-separated INTERNAL-IPs of CP nodes from Phase 2 and explain why the lookup would fail.
After all questions, render the full Package CR to <config-dir>/cozystack-platform-package.yaml and show the file path + content. Offer: Accept / Edit <key> / Cancel. <config-dir> comes from state.config_dir written by cozystack:wizard Phase 1; if invoked directly, ask the operator (default $PWD).
If state.sops.enabled is true, the skill sops --encrypt --in-places the file after Accept and decrypts to a tempfile for the Phase 7 kubectl apply --filename. The encrypted form is what gets committed; the tempfile is removed immediately after apply.
Phase 5 — Plan presentation (STOP GATE 2)
After Accept of the Phase 4 intake values, build the consolidated plan view. This is the final operator confirmation before the skill runs helm install + Platform Package apply + watch loop:
cozystack:cluster-install plan
context: $CTX ($API_URL)
installer release: oci://ghcr.io/cozystack/cozystack/cozy-installer:$INSTALLER_VERSION_OCI (OCI tag = git tag with the v stripped)
installer variant: $INSTALLER_VARIANT
helm release ns: cozy-system (--create-namespace; labeler hook stamps PSA — v1.4+)
platform variant: $PLATFORM_VARIANT
bundles: $BUNDLES_CSV
networking:
podCIDR: $POD_CIDR (matches apiserver: ✓/✗)
podGateway: $POD_GW
serviceCIDR: $SVC_CIDR (matches apiserver: ✓/✗)
joinCIDR: $JOIN_CIDR
kubeovn MASTER_NODES: $MASTER_NODES or "(label lookup)"
apiServerHost: $API_HOST ($API_HOST_SOURCE)
# $API_HOST_SOURCE values:
# "Talos KubePrism" — variant=talos
# "extractedprism (default)" — variant=generic
# "operator override" — --api-host=<ip> or --no-extractedprism
# "managed (hosted)" — variant=hosted, not set
publishing:
host: $HOST (ownership confirmed: ✓ / nip.io)
dns pre-flight: $DNS_PROBE_RESULT (✓ matches external IPs / ⚠ pending / nip.io magic)
apiServerEndpoint: $API_ENDPOINT
exposure mode: $MODE
external IPs: $EXT_IPS
exposed: $EXPOSED_CSV
cert solver: $SOLVER (http01 / dns01)
storage (ZFS):
per-node:
$NODE1: $DEVICES1 → zpool $POOL_NAME ($LAYOUT)
$NODE2: $DEVICES2 → zpool $POOL_NAME ($LAYOUT)
linstor pool: $LINSTOR_POOL_NAME
actions on Continue:
1. Storage provisioning per node (Phase 5.5; one approval per node)
2. (generic only, unless --no-extractedprism) install extractedprism DaemonSet for kube-apiserver HA (~1 min)
3. (if cozy-system namespace exists but unowned) adopt namespace into cozy-system/cozy-installer
4. helm upgrade --install cozy-installer ... --namespace cozy-system --create-namespace (~2 min)
5. wait deploy/cozystack-operator Available; wait CRD packages.cozystack.io Established
6. kubectl apply --filename /tmp/.../platform-package.yaml
7. wait root Tenant CR, patch spec.host + spec.ingress=true (~3 min — required for Phase 8 to ever finish; breaks the OIDC chicken-and-egg)
8. poll HRs every 30s until all Ready=True (~30–60 min)
9. print access summary
options: Continue / Edit values / Cancel
If Skip — pool already managed externally was picked in Phase 4, the storage block is replaced with storage: skipped (externally managed) and Phase 5.5 is omitted from the action list.
Phase 5.5 — Storage provisioning (STOP GATE per node, or batch when identical)
Skip entirely if Phase 2 detected an existing zpool with the name Phase 4 collected and the operator chose reuse — in that case the skill verifies the existing pool with zpool status instead of creating.
Batch-all-identical pattern: when every storage-scope node has the same layout (single device path, same cozystack_intake.storage_pref.layout_per_node[*] value, same zpool name, same distribution), the per-node STOP GATE collapses to one approval that covers all nodes. Three identical CPs each picking /dev/sdb for single layout shouldn't require three identical yes/no rounds. Determine identical-batch shape:
LAYOUTS=$(yq '[.cozystack_intake.storage_pref.layout_per_node[]] | unique' "$STATE_FILE")
DEVICES_PER_NODE=$(yq '[.cozystack.storage.discovered_devices[][] | length] | unique' "$STATE_FILE")
DISTRIBUTIONS=$(yq '[.inventory.nodes[].distribution // .cluster.distribution] | unique' "$STATE_FILE")
if [ "$(jq 'length' <<<"$LAYOUTS")" -eq 1 ] && \
[ "$(jq 'length' <<<"$DEVICES_PER_NODE")" -eq 1 ] && \
[ "$(jq 'length' <<<"$DISTRIBUTIONS")" -eq 1 ]; then
BATCHABLE=1
fi
When BATCHABLE=1, surface a single STOP GATE that covers all nodes:
storage provisioning — $N identical nodes ($DISTRIBUTION)
layout: $LAYOUT (single device per node, same path on each)
per node: /dev/sdb → zpool $POOL_NAME
nodes that will be provisioned:
node0 /dev/sdb → zpool $POOL_NAME (single)
node1 /dev/sdb → zpool $POOL_NAME (single)
node2 /dev/sdb → zpool $POOL_NAME (single)
options:
- Provision all $N (Recommended — identical config across nodes)
- Step through each node — switch to one-STOP-GATE-per-node flow
- Cancel install
On Provision all, run the per-node loop without per-node prompts. On any per-node failure, abort the batch and surface which node failed + the failure mode (do not silently continue with the remaining nodes — partial cluster is harder to debug than no cluster).
For non-identical configurations (mixed layouts: one node single, another mirror; or different device paths discovered per node), drop to the per-node STOP GATE flow below — each node gets its own approval because the operator is making a real choice each time.
For every other node in storage scope, run the per-node loop. The mechanism splits on cluster.distribution:
-
Talos: cannot use kubectl debug node --image=alpine:3 -- chroot /host zpool create. Three reasons it fails:
alpine:3 ships musl libc; the Talos host's /usr/local/sbin/zpool is glibc-linked and depends on /lib64/ld-linux-x86-64.so.2, which doesn't exist on Talos rootfs (Talos itself is musl-statically-linked; the zfs userspace lives only inside the ext-zfs-service namespace).chroot /host /bin/sh — /bin/sh does not exist in Talos rootfs.- The Pod Security Admission
baseline enforced on default and kube-system rejects the privileged debug Pod the sysadmin profile creates.
Use a privileged DaemonSet bootstrap pattern instead — see references/storage-backends.md Talos section for the exact ubuntu+apt+sgdisk+bind-mount commands. Summary: create a cozy-storage-bootstrap namespace with PSA label pod-security.kubernetes.io/enforce=privileged, run a one-shot privileged pod from ubuntu:24.04, apt-get install -y zfsutils-linux, bind-mount /dev/zfs and the chosen disk's host path, sgdisk partition before zpool create (Talos has no udev inside the pod, so zpool create /dev/sdb cannot wait for partition discovery — partition manually first), then zpool create -f $POOL_NAME /dev/sdbN.
-
Ubuntu / k3s / kubeadm: the original kubectl debug node --image=alpine:3 -- chroot /host path works (host has glibc + /bin/sh + zfs userspace pre-installed by ansible-cozystack).
-
Present the exact commands per distribution, with $DEVICE (or $DEVICES for mirror / raidz) and $POOL_NAME substituted. Pull the bodies from references/storage-backends.md.
storage provisioning — node $NODE ($DISTRIBUTION)
layout: $LAYOUT (single / mirror / raidz)
target: $DEVICES → zpool $POOL_NAME
commands (Talos path — privileged DaemonSet from ubuntu:24.04):
# one-shot pod in cozy-storage-bootstrap; PSA=privileged
apt-get install -y zfsutils-linux
sgdisk --zap-all $DEVICE && sgdisk --new=1:0:0 --typecode=1:bf01 $DEVICE
partprobe $DEVICE && sleep 1
zpool create -o ashift=12 $POOL_NAME ${DEVICE}1
zfs set compression=lz4 $POOL_NAME
zfs set atime=off $POOL_NAME
commands (Ubuntu / k3s / kubeadm path — kubectl debug chroot /host):
zpool create -o ashift=12 $POOL_NAME $VDEV_SPEC
zfs set compression=lz4 $POOL_NAME
zfs set atime=off $POOL_NAME
verify (read-only) commands the skill will run afterwards:
zpool status $POOL_NAME
zpool list -H -o name,size,free,health $POOL_NAME
options: Provision this node / Skip this node / Cancel install
$VDEV_SPEC resolves per layout: $DEVICE for single, mirror $DEVICE1 $DEVICE2 for mirror, raidz $DEVICE1 $DEVICE2 $DEVICE3 for raidz. The Talos path is single-device only in v1 of this skill — multi-device layouts on Talos need per-device sgdisk and the pod's bind-mount list grows accordingly; surface as "not yet supported, use Ubuntu route for mirror/raidz on Talos workers".
-
On Provision this node:
-
Talos path:
kubectl --context $CTX get ns cozy-storage-bootstrap >/dev/null 2>&1 || \
kubectl --context $CTX create ns cozy-storage-bootstrap
kubectl --context $CTX label ns cozy-storage-bootstrap \
pod-security.kubernetes.io/enforce=privileged --overwrite
kubectl --context $CTX --namespace cozy-storage-bootstrap run zpool-create-$NODE \
--image=ubuntu:24.04 --restart=Never --overrides='{...}' \
--command -- /usr/local/bin/cozy-zpool-bootstrap.sh "$DEVICE" "$POOL_NAME"
kubectl --context $CTX --namespace cozy-storage-bootstrap wait pod/zpool-create-$NODE \
--for=condition=Ready --timeout=300s
The --overrides payload mounts /dev into the pod, sets nodeName: $NODE so it runs where the disk lives, sets hostPID: true so partprobe is visible to the host kernel, and sets hostNetwork: true because Phase 5.5 runs before Phase 6 installs cozy-installer (which installs Cilium). Without hostNetwork: true, a pod with the default pod-network needs CNI to assign an IP — Cilium isn't up yet, so the pod stays ContainerCreating. Using the host network sidesteps the need for CNI entirely; the bootstrap pod doesn't open any listening ports. See references/storage-backends.md for the verbatim JSON.
-
Ubuntu / k3s / kubeadm path:
kubectl --context $CTX debug node/$NODE \
--image=alpine:3 --profile=sysadmin --quiet --stdin=false --tty=false \
-- chroot /host /bin/sh -c '<commands-from-step-1>'
Capture stdout/stderr. Print them to the operator verbatim. On Talos, the pod's logs are the source.
-
Verify. Run the verify block from step 1. The zpool must show health: ONLINE. If the output does not match (DEGRADED on a fresh single-disk pool = device failure; missing pool entirely = zpool create silently failed), mark the node as failed and surface the diff.
-
On any failure in step 2 or step 3:
storage provisioning failed on $NODE
stage: create | verify
stderr: <captured stderr>
options: Retry / Skip this node / Cancel install (and offer backout commands)
-
On Skip this node — record the decision. The skill excludes this node from the LINSTOR storage-pool registration block inside the Phase 8 watch loop. Warn the operator: a single-node-skipped cluster degrades DRBD replica count.
-
On Cancel install — show backout commands from references/storage-backends.md (zpool destroy + wipefs --all, plus kubectl delete ns cozy-storage-bootstrap on Talos) for every node that was already provisioned in this Phase 5.5 run. The skill does not auto-rollback (operator must own the destructive step).
-
Pre-existing-data check (every distribution). Before zpool create, probe the target device for residual LVM / DRBD / LINSTOR thin-pool state from previous installs:
pvs --noheadings --options vg_name "$DEVICE" 2>/dev/null
dmsetup ls | grep -E "(linstor|drbd|thin)" || true
wipefs --output=label,type "$DEVICE"
If anything is reported, refuse to proceed without an explicit operator confirmation. A talosctl reset does not wipe user disks by default; previous-install state on /dev/sdb is a common cause of zpool create failing with EBUSY or silently producing a DEGRADED pool. Offer the wipe command (vgchange -an; dmsetup remove --force; dd if=/dev/zero of=$DEVICE bs=1M count=10; sgdisk --zap-all $DEVICE; wipefs --all $DEVICE) for the operator to approve and execute, then re-run.
After every storage-scope node is either provisioned or explicitly skipped, persist the per-node mapping to <config-dir>/.state.yaml under cozystack.storage.nodes[]. The LINSTOR storage-pool registration is not declarative on the ZFS path (the CRD has no zfsPool slot); the skill runs the registration inside the Phase 8 watch loop as soon as linstor-controller reports Ready — see Phase 8 below for the implementation, not a forward-reference.
Phase 5.6 — Install extractedprism DaemonSet (generic HA)
Skip entirely when:
- installer variant is
talos — KubePrism is built-in.
- installer variant is
hosted — provider handles kube-apiserver HA.
- operator passed
--no-extractedprism — skill respects the override and uses --api-host=<ip> instead.
For generic variant without opt-out, install the chart before cozy-installer so the operator's apiServerHost: 127.0.0.1 + apiServerPort: 7445 already resolves when cozystack-operator starts dialing kube-apiserver:
CP_IPS=$(kubectl --context $CTX get nodes \
--selector node-role.kubernetes.io/control-plane \
--output jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}')
ENDPOINTS=$(printf '%s\n' "$CP_IPS" | awk 'NF{printf "%s%s:6443", sep, $0; sep=","}')
cat > "$CONFIG_DIR/extractedprism-values.yaml" <<EOF
endpoints: "${ENDPOINTS}"
EOF
helm --kube-context $CTX upgrade --install extractedprism \
oci://ghcr.io/lexfrei/charts/extractedprism \
--version 0.2.0 \
--namespace kube-system \
--values "$CONFIG_DIR/extractedprism-values.yaml" \
--wait --timeout 5m
Verify the DaemonSet covers every node and pods are Ready (port 7446 health check) before continuing:
kubectl --context $CTX --namespace kube-system rollout status daemonset/extractedprism --timeout=5m
kubectl --context $CTX --namespace kube-system get pods --selector app.kubernetes.io/name=extractedprism --output wide
If any node lacks the extractedprism pod (taint, nodeSelector skip), Cilium on that node will fail to reach the apiserver after install. Surface and ask: continue / re-apply with broader toleration / abort.
extractedprism is a per-node TCP load balancer that listens on 127.0.0.1:7445 and proxies to a healthy CP endpoint with TCP health checks. Source: https://github.com/lexfrei/extractedprism. License: BSD-3-Clause. The chart at oci://ghcr.io/lexfrei/charts/extractedprism ships a DaemonSet with hostNetwork: true, priorityClassName: system-node-critical, and a catch-all toleration so it runs on every node regardless of taints — that is intentional, the proxy is critical infrastructure.
Phase 6 — Install operator
Namespace adoption first if cozy-system exists and lacks Helm metadata (see references/values-template.md). If it's owned by another release, refuse — do not relabel.
OCI registry version format: cozystack publishes the cozy-installer chart to ghcr.io/cozystack/cozystack/cozy-installer with tags like 1.3.3 (no v prefix). If state.cozystack.installer_version carries the canonical vX.Y.Z form (from cozystack/cozystack git tags), strip the v before passing to --version. Phase 5 plan presentation already shows the normalised version so the operator sees what gets run.
INSTALLER_VERSION_OCI="${INSTALLER_VERSION#v}"
helm --kube-context $CTX upgrade --install cozy-installer \
oci://ghcr.io/cozystack/cozystack/cozy-installer \
--version "$INSTALLER_VERSION_OCI" \
--namespace cozy-system --create-namespace \
--set cozystackOperator.variant=$INSTALLER_VARIANT \
--set cozystack.apiServerHost=$API_HOST \
--set cozystack.apiServerPort=$API_PORT \
--wait --timeout 10m
For a v1.3.x install the form is --namespace kube-system with NO --create-namespace (the v1.3 chart templates the namespace itself). See references/values-template.md for both forms side by side — pick the one matching installer_version.
For talos and hosted, drop cozystack.apiServerHost / apiServerPort if not required by the chart.
Verify:
kubectl --context $CTX --namespace cozy-system wait deploy/cozystack-operator \
--for=condition=Available --timeout=300s
kubectl --context $CTX wait crd/packages.cozystack.io \
--for=condition=Established --timeout=300s
If the helm install fails, show events + the operator's logs and stop. Don't retry blindly.
Phase 7 — Apply Platform Package
if [ "$(yq '.sops.enabled // false' "$CONFIG_DIR/.state.yaml")" = "true" ]; then
tmp=$(mktemp); sops --decrypt "$CONFIG_DIR/cozystack-platform-package.yaml" > "$tmp"
kubectl --context $CTX apply --filename "$tmp"; rm -f "$tmp"
else
kubectl --context $CTX apply --filename "$CONFIG_DIR/cozystack-platform-package.yaml"
fi
Confirm the Package was created:
kubectl --context $CTX get package cozystack.cozystack-platform --output yaml
Phase 8 — Watch HelmReleases until green (with inline root Tenant patch)
See references/helmrelease-monitoring.md for the full polling pattern and stuck-state diagnostics.
This phase merges what used to be Phase 7.5 (root Tenant ingress patch) into the watch loop. The motivation: on a real install the tenants.apps.cozystack.io CRD can take longer to install than the wait deadline of a separate phase — sometimes the CRD doesn't exist until after most HRs are Ready, so a fixed 5-minute wait-for-Tenant expires before the CRD lands. Folding the patch into the watch loop makes it event-driven: as soon as the CR appears, the skill patches it and continues monitoring, regardless of where Phase 8 is in its own progression.
Why the patch is needed: cozystack's dashboard ships gatekeeper (oauth2-proxy) which, on startup, does OIDC discovery against the public FQDN https://keycloak.${HOST}/realms/cozy/.well-known/openid-configuration — not an in-cluster service. Without the root ingress controller running, nothing listens on 443, gatekeeper CrashLoopBackOffs, the cozy-dashboard/dashboard HR sits in Unknown: Running 'install' action with timeout of 10m0s and then InstallFailed: context deadline exceeded, Flux remediates and retries forever. cozy-fluxcd/flux-plunger has a hard dependency on cozy-dashboard/dashboard and stays False: dependency is not ready. The phase would never go green.
The dashboard requires OIDC/Keycloak — it is not optional on the supported path. The "Why the patch is needed" note above describes gatekeeper doing OIDC discovery against Keycloak. But Keycloak only deploys when authentication.oidc.enabled: true in the Platform Package — and that key defaults to false, with the isp-full* overlays NOT turning it on. If the skill enables ingress + sets host but never enables OIDC, the result on v1.4.2 is: no cozy-keycloak namespace, no Keycloak HR, and the dashboard falls back to its non-OIDC token-proxy container, which is broken on v1.4.2 — the container starts, never binds :8000 (connection refused, zero logs), and is killed by its own /ping liveness probe every ~45 s → CrashLoopBackOff → the cozy-dashboard/dashboard HR fails install → flux-plunger and the rest of the chain hang. The cluster reports 88/90 HRs Ready and looks "almost done" forever.
So for a usable dashboard the skill must enable OIDC. This is exactly what cozystack's own e2e (hack/e2e-install-cozystack.bats) does — patch the root tenant host, then enable OIDC and expose Keycloak:
kubectl --context $CTX patch package cozystack.cozystack-platform --type merge \
--patch '{"spec":{"components":{"platform":{"values":{"authentication":{"oidc":{"enabled":true}}}}}}}'
kubectl --context $CTX patch package cozystack.cozystack-platform --type merge \
--patch '{"spec":{"components":{"platform":{"values":{"publishing":{"exposedServices":["api","dashboard","keycloak"]}}}}}}'
Better: bake both into the Platform Package CR written in Phase 4/7 from the start (authentication.oidc.enabled: true and keycloak in exposedServices) so there is no second reconcile. The Phase 4 intake should collect a dashboard auth decision — OIDC/Keycloak (recommended; the only path with a working dashboard on 1.4.2) vs none (API-only, no web dashboard) — and only enable OIDC when the operator wants the dashboard. When OIDC is enabled, Keycloak needs a working LE cert for keycloak.${HOST}, so the same DNS/port-80 preconditions as the dashboard host apply (Phase 4 publishing gate already covers this — just make sure keycloak.${HOST} is inside the wildcard).
Skip the root-Tenant patch entirely on isp-hosted or when the system bundle was disabled in Phase 4 — there is no root Tenant CR in those modes.
Watch loop (per 30 s poll):
if kubectl --context $CTX --namespace tenant-root get tenants.apps.cozystack.io root \
--output jsonpath='{.metadata.name}' 2>/dev/null | grep -q '^root$'; then
CUR_INGRESS=$(kubectl --context $CTX --namespace tenant-root get tenants.apps.cozystack.io root \
--output jsonpath='{.spec.ingress}')
CUR_HOST=$(kubectl --context $CTX --namespace tenant-root get tenants.apps.cozystack.io root \
--output jsonpath='{.spec.host}')
if [ "$CUR_INGRESS" != "true" ] || [ "$CUR_HOST" != "$HOST" ]; then
kubectl --context $CTX --namespace tenant-root patch tenants.apps.cozystack.io root \
--type=merge --patch "{\"spec\":{\"ingress\":true,\"host\":\"${HOST}\"}}"
echo "patched tenants/root spec.host=${HOST} ingress=true at $(TZ=UTC date -Iseconds)"
fi
fi
kubectl --context $CTX get hr --all-namespaces \
--output jsonpath='{range .items[?(@.status.conditions[?(@.type=="Ready" && @.status!="True")])]}{.metadata.namespace}/{.metadata.name} {end}'
On the full system-bundle path you may also want the root tenant's etcd/monitoring/seaweedfs services (this is what cozystack's own hack/e2e-install-cozystack.bats patches): extend the patch to {"spec":{"ingress":true,"host":"${HOST}","monitoring":true,"etcd":true,"seaweedfs":true}} when those were selected in Phase 4. Leave them at their defaults otherwise.
HelmRelease $NS/$NAME has been Failing for $T minutes.
Last condition: <message>
options: Keep waiting / Capture diagnostics and pause / Abort
Capture diagnostics runs the bundle script from references/issue-templates.md.
STOP GATE 3 — All HRs Ready AND storage pools registered
Don't print success until both conditions hold:
HR_NOT_READY=$(kubectl --context $CTX get hr --all-namespaces \
--output jsonpath='{range .items[?(@.status.conditions[?(@.type=="Ready" && @.status!="True")])]}{.metadata.namespace}/{.metadata.name} {end}')
EXPECTED_POOLS=$(yq '.cozystack.storage.nodes | length' "$STATE_FILE")
if [ "$EXPECTED_POOLS" -gt 0 ]; then
ACTUAL_POOLS=$(kubectl --context $CTX --namespace cozy-linstor exec deploy/linstor-controller -- \
linstor storage-pool list --output-version v1 2>/dev/null \
| jq --raw-output '[.[] | select(.provider_kind == "ZFS") | .node_name] | unique | length')
else
ACTUAL_POOLS=0
EXPECTED_POOLS=0
fi
[ -z "$HR_NOT_READY" ] && [ "$ACTUAL_POOLS" -eq "$EXPECTED_POOLS" ]
Why both: an HR can report Ready=True (from Flux's lifecycle: "install action succeeded") before the underlying Deployment has any ready replicas. The LINSTOR-controller HR is the canonical case — Ready=True at the moment the helm release lands, but kubectl get deploy linstor-controller --output jsonpath='{.status.readyReplicas}' is still empty for another 30–60 s while the pod starts. Exiting on HR-Ready-only races the inline storage-pool registration block: the watch loop sees "all HRs Ready" and exits before the per-poll registration check has fired against a ready linstor-controller, leaving pools unregistered. Operator then has to register them manually after cluster-install already declared success.
The combined gate avoids that race without re-introducing the post-watch deadlock (paas / monitoring HRs that depend on PVCs would never reach Ready if registration ran after all HRs were required to be Ready — see references/known-failures.md).
LINSTOR storage-pool registration — inline (gated on linstor-controller Ready)
Why this is folded into the watch loop rather than a post-watch phase: any HR in the paas or monitoring bundles that requests a PVC stays Pending until LINSTOR has a registered storage pool for at least one node. If pool registration runs after "all HRs Ready", the watch loop can never exit — Phase 8 deadlocks waiting for HRs whose PVCs deadlock waiting for the pool. This is the same shape as the root-Tenant ingress patch: the right gate is "the producer became Ready", not "everything is Ready".
The same watch loop documented above is extended with a storage-pool registration block, gated on linstor-controller Ready, not on all-HR-Ready:
LINSTOR_READY=$(kubectl --context $CTX --namespace cozy-linstor get deploy linstor-controller \
--output jsonpath='{.status.readyReplicas}' 2>/dev/null)
if [ "${LINSTOR_READY:-0}" -ge 1 ]; then
yq --output-format=json '.cozystack.storage.nodes' "$STATE_FILE" \
| jq --compact-output '.[]' \
| while IFS= read -r entry; do
NODE=$(jq --raw-output '.name' <<<"$entry")
ZPOOL=$(jq --raw-output '.zpool' <<<"$entry")
LINPOOL=$(jq --raw-output '.linstor_pool' <<<"$entry")
if kubectl --context $CTX --namespace cozy-linstor exec deploy/linstor-controller -- \
linstor storage-pool list --node "$NODE" --storage-pool "$LINPOOL" \
--output-version v1 2>/dev/null | grep -q "$LINPOOL"; then
continue
fi
kubectl --context $CTX --namespace cozy-linstor exec deploy/linstor-controller -- \
linstor storage-pool create zfs "$NODE" "$LINPOOL" "$ZPOOL" \
|| { echo "storage-pool register failed on $NODE"; exit 1; }
done
fi
On per-node registration failure, the watch loop aborts Phase 8 and writes failed_at: "linstor-storage-pool" with error_detail containing the failing node and the linstor storage-pool create stderr. Partial registrations are kept (idempotent retry-friendly); operator can re-invoke after fixing the root cause.
Verification (after all HRs Ready, separately from registration):
kubectl --context $CTX --namespace cozy-linstor exec deploy/linstor-controller -- \
linstor storage-pool list | grep -E "zfs|node"
Phase 8.6 — Default StorageClasses
Gate on the live cluster, not on a version number. Earlier guidance skipped this phase on installer_version ≥ 1.4.0 on the assumption that the tenants.apps.cozystack.io CRD exposes spec.storageClasses and the operator creates the StorageClasses from the tenant declaration. That assumption is false on at least v1.4.2 — the shipped tenant CRD has no storageClasses field (kubectl get crd tenants.apps.cozystack.io -o yaml | grep -c storageClass → 0), nothing auto-creates StorageClasses, and the cluster reaches "all HRs Ready" with kubectl get storageclass empty. Every stateful workload (keycloak-db, etcd, seaweedfs, vmstorage/vlstorage) then sits in Pending: unbound immediate PersistentVolumeClaims, which cascades: keycloak CrashLoops with no DB → cozystack-api/controller/dashboard never go Ready.
So the correct gate is a live check, not a version branch:
EXISTING_SC=$(kubectl --context $CTX get storageclass --output name 2>/dev/null | wc -l | tr -d ' ')
if [ "$EXISTING_SC" -gt 0 ]; then
echo "StorageClasses already present — skip (operator or a future chart created them):"
kubectl --context $CTX get storageclass
else
echo "No StorageClasses — applying linstor defaults (see manifest below)."
fi
If a future cozystack release does start auto-creating StorageClasses, the live check skips this phase automatically — no version bump to the skill needed. Until then, the skill creates them on every version where the cluster comes up empty.
The skill writes two StorageClasses:
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: linstor.csi.linbit.com
parameters:
linstor.csi.linbit.com/storagePool: "${LINSTOR_POOL_NAME}"
linstor.csi.linbit.com/placementCount: "1"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: replicated
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: linstor.csi.linbit.com
parameters:
linstor.csi.linbit.com/storagePool: "${LINSTOR_POOL_NAME}"
linstor.csi.linbit.com/placementCount: "3"
linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
kubectl --context $CTX apply --filename "$CONFIG_DIR/storageclasses-default.yaml"
kubectl --context $CTX get storageclass
Timing — create the StorageClasses inside the Phase 8 watch loop, not after it. Apply them as soon as local/replicated are absent and the LINSTOR pools are registered (same gate as the inline pool registration), NOT after "all HRs Ready". stateful HRs in the paas/monitoring bundles (keycloak, etcd, seaweedfs, vmstorage) request PVCs that stay Pending until a default StorageClass exists, so an "all-HRs-Ready → then create SCs" ordering deadlocks the watch loop the same way the LINSTOR pool registration would. Folding SC creation into the loop (gated on linstor-controller Ready) lets those PVCs bind and the dependent HRs converge. One subtlety: a PVC created with no storageClassName before a default SC exists records storageClassName: "" and will NOT retroactively pick up a later default — but every cozystack chart pins storageClassName explicitly (replicated), so in practice the Pending PVCs bind as soon as the named class appears. If you do hit a genuinely class-less Pending PVC, it must be recreated after the default exists.
replicated is marked as the default; local is a single-replica fallback for system workloads that don't need replication. On clusters with fewer than 3 storage-providing nodes, drop placementCount for replicated to match — the skill auto-derives this from cozystack.storage.nodes[] count.
Phase 9 — Post-install verification
kubectl --context $CTX get hr --all-namespaces | grep -v ' True ' || echo "all HRs Ready"
kubectl --context $CTX get pods --all-namespaces --field-selector=status.phase!=Running,status.phase!=Succeeded
Confirm the root Tenant patch (applied inline by Phase 8) actually landed:
kubectl --context $CTX --namespace tenant-root get tenants.apps.cozystack.io root \
--output jsonpath='{.spec.ingress}'
Spot-check dashboard ingress and certificate:
kubectl --context $CTX get ingress --all-namespaces | grep dashboard
kubectl --context $CTX get certificate --all-namespaces | grep -E 'dashboard|keycloak'
If certificates are still Ready=False after 2 minutes, do not advise the operator to wait longer — "first issuance takes ~5 min" is a misleading mitigation that hides the actual failure mode. On a healthy install, an ACME HTTP-01 challenge resolves in 10–30 seconds; anything past 2 minutes means the challenge is stuck on a definite cause (rate limit, DNS unreachable, port 80 firewalled, externalIPs mismatch, wildcard config). Inspect the ACME Challenge CRs and the cert-manager log directly:
kubectl --context $CTX get challenges --all-namespaces \
--output custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATE:.status.state,REASON:.status.reason,DOMAIN:.spec.dnsName
kubectl --context $CTX get orders --all-namespaces
kubectl --context $CTX --namespace cozy-cert-manager logs deploy/cert-manager --tail=100 \
| grep -iE 'acme|challenge|order'
Common stuck states and where to look:
Challenge.status.state | Reason in log | Likely cause | Fix |
|---|
pending | Waiting for HTTP-01 challenge propagation | DNS not resolving challenge host | Configure wildcard A-record at the registrar |
pending | dial tcp ...:80: i/o timeout | port 80 not reachable from LE validators | Open port 80 at firewall / LB |
pending | connect: connection refused (RST) | externalIPs on a NAT'd provider (OCI / GCP NAT / AWS EIP) match wrong address | Re-pick external_ips.strategy=internal — see Phase 4 |
invalid | urn:ietf:params:acme:error:rateLimited | Let's Encrypt 5-cert-per-week rate limit hit | Wait 7 days, or switch to staging issuer, or dns01 solver |
invalid | incorrect response from authoritative server | wildcard A-record pointing at the wrong IP | Re-verify which IPs cozystack actually advertises with kubectl get svc -n tenant-root-ingress |
valid but Certificate not Ready | (none — challenge done) | Order Issuing slow (usually <5s) | Wait one more minute; if still stuck, kubectl describe certificate for the precise error |
For nip.io hosts: A 2-minute pending challenge is itself anomalous because nip.io DNS is instantaneous. Suspect port 80 firewalling or wrong external IPs first.
That is the operator's fix, not the skill's — DNS / firewall configuration is outside cluster scope. The skill surfaces the diagnosis without speculating on the cause.
Phase 9.1 — End-to-end reachability probe (mandatory final gate)
"All HelmReleases Ready" is a cluster-side signal — it reports the lifecycle of helm-controller, not whether the dashboard is actually reachable from outside the cluster. Real-world failure modes (OCI 1:1 NAT externalIPs mismatch, DNS not yet propagated, ingress controller running on the wrong nodes) routinely produce Ready=True HRs with an unreachable dashboard. The skill must verify reachability before declaring success, otherwise the next operator action is "click the dashboard URL → 530 / RST → debug from scratch", which is the worst possible failure UX.
From the operator's workstation (not from inside the cluster):
EXPECTED_IPS=$(jq -r '.publishing.external_ips[]' <<< "$STATE_JSON" | sort -u)
RESOLVED=$(dig +short "dashboard.${HOST}" | sort -u)
curl --silent --output /dev/null --write-out '%{http_code} %{exitcode}\n' \
--connect-timeout 5 --max-time 10 \
--insecure \
"https://dashboard.${HOST}/"
Pass conditions (any of):
- HTTP 200 / 302 / 401 — dashboard answered (401 is the gatekeeper redirect; that's fine).
- HTTP 200 from
https://api.${HOST}:6443/healthz with --insecure — apiserver ingress is up.
Fail conditions (any of these is failed_at, not a successful install):
curl exit 6 (Couldn't resolve host) — DNS not configured. On nip.io this should never happen; on custom-fqdn it means the wildcard A-record isn't published yet. Cross-reference: RESOLVED (will be empty).curl exit 7 (Couldn't connect to host) — covers both ECONNREFUSED (TCP RST — nothing listening at that IP/port) and EHOSTUNREACH / ENETUNREACH (no route to the IP). They look identical at the curl layer; distinguish by checking ip route get <EXPECTED_IP> from the workstation:
- Route returns
unreachable → no path; workstation can't actually reach the cluster (firewall, VPN not up).
- Route returns a normal egress → RST; almost always externalIPs misconfig on a NAT'd provider. Cross-reference:
cozystack_intake.external_ips.strategy and Node.status.addresses.
curl exit 28 (Timeout) — packets reach the destination IP but no SYN/ACK comes back. Usually port 443 firewalled at the cloud-provider security group level; can also be wrong IPs in DNS (EXPECTED_IPS ≠ RESOLVED).- HTTP 530 (Cloudflare-style "origin unreachable") — DNS points through a CDN to wrong upstream.
- HTTP 502 / 503 — ingress is up but backend isn't ready; usually transient, retry once after 30 s. Persistent 502/503 (over 2 min) → real failure.
When the probe fails, write failed_at: "external-reachability" with error_detail containing the curl output, the resolved-vs-expected IP set, and the Node.status.addresses mismatch (if any). The cluster stays as-is — no rollback — and cozystack:debug gets dispatched by the wizard.
For sandbox / nip.io installs where the operator deliberately picked a non-routable address (192.168.x.x on a home LAN where the workstation can reach the cluster), the operator can pass --skip-external-reachability to downgrade the probe to a warning. The skill prints the warning and still writes completed_at, but the Phase 10 NOTES carry a "external reachability not verified — operator opted out" line so the result is auditable.
Phase 9.2 — Write status.cluster-install (mandatory)
Before printing Phase 10 NOTES, the skill must write exactly one of completed_at / failed_at to <config-dir>/.state.yaml. This is the contract cozystack:wizard Phase 5 dispatch loop relies on — without it the chain hangs at the next dispatch decision (no completed_at ⇒ no progression; no failed_at ⇒ no cozystack:debug auto-dispatch).
Success path (every gate from Phase 8 + Phase 9 passed):
status:
cluster-install:
dispatched_at: <existing-ISO-from-wizard>
completed_at: <ISO_TS_UTC_NOW>
installer_version: <INSTALLER_VERSION>
platform_variant: <PLATFORM_VARIANT>
bundles: [...]
helm_releases: { ready: <N>, total: <N> }
Failure path (any STOP GATE refused, any Phase 8 HR stuck past timeout, any Phase 9 verification failed):
status:
cluster-install:
dispatched_at: <existing-ISO-from-wizard>
failed_at: <ISO_TS_UTC_NOW>
error: "<one-line summary; what gate / which HR / which check>"
error_detail: "<optional multi-line; kubectl describe output, log excerpt>"
Write with sops --encrypt --in-place afterwards when state.sops.enabled: true. Never write both fields. Never leave both unset.
Phase 10 — NOTES-style access summary
Lean. Entry points, file paths, where credentials live, where to read more. No prose paragraphs.
cozystack ready
cluster:
context: $CTX
api: $API_URL
installer: oci://ghcr.io/cozystack/cozystack/cozy-installer:$VERSION
installer variant: $INSTALLER_VARIANT
platform variant: $PLATFORM_VARIANT (bundles: $BUNDLES_CSV)
nodes: $N ready | helmreleases: $M / $M ready
access:
dashboard: https://dashboard.$HOST
api: $API_ENDPOINT
grafana: https://grafana.$HOST (if monitoring HR ready)
cert solver: $SOLVER (http01 / dns01)
tls certs: $CERT_READY_COUNT / $CERT_TOTAL Ready (cert-manager — first issuance can take ~5 min)
storage (ZFS):
per-node: $N × zpool '$POOL_NAME' ($LAYOUTS)
linstor: storage pool '$LINSTOR_POOL_NAME'
verify: kubectl --context $CTX --namespace cozy-linstor exec deploy/linstor-controller -- linstor storage-pool list
api server HA:
apiServerHost: $API_HOST ($API_HOST_SOURCE)
# On generic with extractedprism (default), every node dials 127.0.0.1:7445 and the DaemonSet
# forwards to a healthy CP endpoint. To remove extractedprism later:
# helm --kube-context $CTX uninstall extractedprism --namespace kube-system
# (and re-apply the cozy-installer chart with the operator's chosen apiServerHost).
verify: kubectl --context $CTX --namespace kube-system get daemonset extractedprism 2>/dev/null \
|| echo "extractedprism not installed (Talos KubePrism / operator override / hosted variant)"
credentials:
keycloak admin (initial):
kubectl --context $CTX --namespace cozy-keycloak get secret keycloak-credentials \
--output jsonpath='{.data.admin-password}' | base64 --decode; echo
dashboard sso: SSO via Keycloak — first user provisioned by tenant-root
artifacts on disk:
values file: <config-dir>/cozystack-platform-package.yaml
helm release: cozy-system/cozy-installer
cluster-scoped: package.cozystack.io/cozystack.cozystack-platform
handy commands:
watch state: kubectl --context $CTX get hr --all-namespaces
operator logs: kubectl --context $CTX --namespace cozy-system logs deploy/cozystack-operator -f
list tenants: kubectl --context $CTX get tenants.apps.cozystack.io --all-namespaces
next:
docs: https://cozystack.io/docs/v<X.Y>/
first tenant: https://cozystack.io/docs/v<X.Y>/getting-started/install-cozystack#51-setup-root-tenant-services
upgrade later: run cozystack:cluster-upgrade
Fill all $placeholders from collected values and live lookups. Drop lines whose preconditions weren't met (e.g. omit grafana line when the paas bundle was not enabled).
Error handling — upstream issue handoff
If any phase hits a fatal failure that looks like an upstream bug or doc gap, follow references/issue-templates.md:
- Stop the current phase. Don't try further fixes.
- Run the diagnostic-bundle script. Show the bundle path.
- Pick the right repo from the routing table.
- Render the matching issue-body template, fill placeholders from collected state, write to
/tmp/.../issue-body.md. Show the rendered text to the user.
- Print the exact
gh issue create command but do not execute it.
Guardrails
- NEVER apply, patch, label, install, or delete without showing the exact command and getting explicit user approval. Prior approval does not carry forward.
- NEVER overwrite values, namespaces, or labels that belong to another Helm release. Refuse, surface ownership, and let the user decide.
- NEVER skip a STOP GATE because earlier gates passed cleanly.
- NEVER assume current
kubectl context is the right one — pin --context in every command, every time.
- NEVER call success while any HR is not Ready, even on a 59-minute clock.
- NEVER open a GitHub issue automatically. Draft, show, hand the command to the user.
- NEVER include private infrastructure names (cluster name, client, internal namespaces) in drafted public issue bodies. Replace with generic placeholders.
- NEVER create a storage pool without explicit per-node approval in Phase 5.5 — even if the operator approved the consolidated plan in Phase 5. Plan approval does not cascade into mutations.
- NEVER
zpool create over an existing pool. If zpool list shows the target name already exists, refuse and ask the operator to either pick a different name, reuse the existing pool, or destroy it manually with zpool destroy + wipefs (references/storage-backends.md backout section).
- NEVER offer LVM / LVM-thin as a storage backend. Cozystack standardises on ZFS; LVM paths were removed because cozystack does not validate or document them. Operators who need LVM are on their own with the piraeus-operator CRD.
- NEVER bootstrap Talos nodes or invoke
boot-to-talos / talm from inside this skill — that flow lives in /cozystack:talos-bootstrap. Refuse and hand off.
- NEVER auto-rollback a partially provisioned storage state — print backout commands and let the operator decide.
- NEVER accept a custom
publishing.host without an explicit operator confirmation that they own the domain and will configure wildcard DNS — the HTTP-01 cert solver fails silently otherwise. nip.io patterns skip this gate because nip.io is publicly hosted DNS.
- ALWAYS patch
tenants/root with BOTH spec.host=$HOST and spec.ingress=true from inside the Phase 8 watch loop as soon as the CR appears, on system-bundle installs. The root tenant ships with spec.host: "" and does not inherit publishing.host, so an ingress-only patch leaves every per-tenant ingress object rendering against an empty domain. The OIDC chicken-and-egg makes Phase 8 unreachable otherwise — dashboard / keycloak / flux-plunger loop forever, every other downstream HR stalls on the missing root ingress. The CR can appear at any point during the watch loop; do not gate the patch behind a fixed pre-Phase-8 wait.
- ALWAYS read variant overlays and
requirements.md before declaring "this looks fine" — variant-specific checks (CP-label value, ZFS availability, KubeOVN MASTER_NODES) are easy to miss.
- ALWAYS pull live data over cached assumption:
kubectl get over "I think this is …".
- ALWAYS write Phase 4 collected values to disk in
<config-dir>/cozystack-platform-package.yaml before applying — the file is part of the diagnostic bundle if Phase 8 fails. ZFS pool registration is stored separately under <config-dir>/.state.yaml cozystack.storage.nodes[] and replayed by the Phase 8 post-Ready hook (there is no LinstorSatelliteConfiguration CR for the ZFS path).
- NEVER silently fall back to plain writes when
state.sops.enabled is true but sops is missing. Refuse Phase 4 and tell the operator to install sops + the configured age/PGP key, or re-invoke with --no-sops.
- NEVER leave plain
<config-dir>/cozystack-platform-package.yaml on disk between Phase 4 and Phase 7 when sops is on. Decrypt to a temp file for kubectl apply, remove the temp file immediately after.
References
references/requirements.md — cluster + node + variant matrix.references/node-checks.md — exact kubectl debug node commands and value thresholds.references/variants.md — picker logic and what each variant deploys.references/values-template.md — canonical chart values, Platform Package YAML, extractedprism values shape, and the ZFS pool registration hook.references/storage-backends.md — ZFS pool create / verify / backout commands run in Phase 5.5. Cozystack standardises on ZFS; LVM paths are explicitly out of scope (the doc spells out why). Talos privileged DaemonSet bootstrap pattern lives here.references/provider-pitfalls.md — provider-specific networking and runtime gotchas: OCI 1:1 NAT, GCP NAT'd external IPs, AWS EIP / NLB proxy-protocol, Talos system-extension namespacing, PSA baseline on k8s 1.25+, talosctl reset preserving user disks, cozystack v1.3.x quirks. Cross-referenced from Phase 4 publishing slot and Phase 5.5 storage.references/helmrelease-monitoring.md — polling pattern, stuck-state triage.references/known-failures.md — top failures with recovery commands.references/issue-templates.md — diagnostic bundle script and issue-body templates per upstream repo.
External:
https://cozystack.io/docs/v1.3/install/kubernetes/generic/ — primary install guide.https://cozystack.io/docs/v1.3/install/hardware-requirements/ — hardware sizing.https://github.com/cozystack/cozystack — chart, operator, packages.https://github.com/cozystack/ansible-cozystack — node-prep playbooks.
