cpk-operator

name	cpk-operator
description	Bring up, move, recover, and tear down container packages running on a fleet of machines. Trigger when the user wants to deploy a host, migrate it between machines or providers, fail over to a replica, snapshot/restore data, walk the fleet dependency graph, or troubleshoot a running deployment.

cpk-operator skill

The administration / operation companion to cpk-author. Author the package once; operate it across the fleet from here. Per FEAT-323.

1. Design principles

• Educational. The verb surface mirrors the model: cpk hosts <verb> walks the host lifecycle; cpk machines <verb> walks the inventory; cpk provider <verb> walks the cloud bindings. Each libexec/cpk/<verb> is a single file that reads end-to-end.
• Functional. No control plane and no stored resource state. Every operator command is idempotent: rerun until it converges, rerun after a failure, rerun to verify. The fleet inventory lives in the live cloud, queried each invocation.
• Decentralized. A host belongs to its master machine; replicas mirror; failover is "promote a replica". No central directory.
• Simple. Operations compose from hosts, machines, provider, ip, dns, cost. No hidden side effects.

2. The model an operator sees

Verb	Substrate	What it controls
cpk provider	provider registry	drivers (local-ssh, hetzner, fly, aws), credentials, capabilities
cpk machines	inventory	live list across providers; online, exec, adopt, create, destroy, volume-snapshot
cpk ip	public IPs	per-provider IP query / claim / release
cpk dns	DNS records	per-provider zone read + record upsert
cpk cost	cost rollup	cross-provider spend, --by org / --by host
cpk hosts	host deployments	up / down / migrate / start / stop / drain / resume / test / status / sync / split / frontend / edge / zone / graph / snapshot / snapshot-diff / snapshot-export / volume-snapshot / rollback / seed-data / seed-secrets / clone-org / promote-org / destroy-org
cpk container	container packages	build, run, data-home, volumes, depends, persist, update, pull, push

3. Operator journeys

3.1. Bring a host up on a fresh cloud VM

cpk provider add hetzner             # one-time: pick a driver
secret set hetzner:token             # one-time: API token
cpk provider add hetzner-dns         # for the org's DNS zone
secret set hetzner-dns:token

cpk hosts up web.acme --provider hetzner --type cx22

up provisions a labelled VM, registers it as srv@<ip>, attaches the host, builds + pushes the cpk, upserts a DNS A record, starts the container. Re-runnable: each step converges. If the host carries .cpk/config/machine-type (written by clone-org --machine-type), up honours it as the default --type.

3.2. Move a host between machines (single provider)

cpk hosts add    web.acme spare-01   # attach to a second machine
cpk hosts master web.acme spare-01   # promote (flips data-sync direction)
cpk hosts start  web.acme            # runs on the new master
cpk hosts del    web.acme old-01     # retire the old binding

3.3. Migrate across providers (FEAT-314 + FEAT-324)

cpk hosts migrate web.acme --provider fly --type shared-cpu-1x

migrate ensures the target machine, brings the data across, cuts over master + DNS, and retires the source. Cross-provider (e.g. hetzner → fly) goes through FEAT-324's contract: snapshot pull from source, image build + volume populate on target.

Two flags for the upgrade-pipeline use cases:

cpk hosts migrate web.acme --provider fly --with-dependents
cpk hosts migrate web.acme --provider fly --preserve-wg

• --with-dependents drains consumers in reverse-topo order, runs the migrate, then resumes them in inverse drain order. Migration failure leaves consumers drained for manual recovery.
• --preserve-wg backs up the source's wireguard identity before transport and restores it on the target after the cut-over -- mesh consumers stay pointed at the same cryptographic peer.

3.4. Frontend an on-prem workload

cpk hosts frontend web.acme \
  --provider hetzner \
  --upstream 10.7.0.42 \
  --ports 80,443

Provisions a cloud VM as a public entry, forwards the listed ports to the workload's mesh address via the cluster VPN, updates DNS. The workload itself stays on-prem.

3.5. Snapshot, rollback, and inspection (FEAT-321 + FEAT-327)

cpk hosts snapshot web.acme file backup       # files snapshot
cpk hosts snapshot web.acme postgres backup   # logical dump
cpk hosts snapshot web.acme list              # ids
cpk hosts rollback web.acme --to <id>         # drain + restore + resume

The snapshot proxy honours per-host retention: a .cpk/config/retention <n> file pushes that value to the remote as max-backups before any backup, so the ring prunes per-host without a fleet-wide tweak.

Read-only inspection:

cpk hosts snapshot-diff a.acme b.acme              # file diff
cpk hosts snapshot-diff a.acme b.acme --handler=postgres  # DDL diff

Encrypted offsite handoff:

cpk hosts snapshot-export web.acme --pass-file ~/.cpk/snap.pass \
    > /backup/web.acme.$(date +%FT%T).enc

For image-only providers (fly.io) the in-VM snapshot tool isn't reachable; use the provider-native verb:

cpk hosts volume-snapshot web.acme list
cpk hosts volume-snapshot web.acme create --volume <vol-id>
cpk hosts volume-snapshot web.acme restore <snapshot-id>

3.6. Walk the fleet's dependency graph (FEAT-320 / FEAT-317)

cpk container depends web.acme                # direct
cpk container depends --transitive web.acme   # full closure
cpk hosts graph example.com                   # topo-sorted DAG
cpk hosts graph example.com --dot | dot -Tsvg # graphviz
cpk hosts zone web.acme                       # public / dmz / int

cpk hosts up <root> --with-dependents brings the closure up in graph order; cpk hosts down <root> --with-dependents tears down in reverse. Cycles are fatal.

3.7. Failover to a replica

cpk hosts machines web.acme          # list bindings
cpk hosts master   web.acme spare-01 # promote
cpk hosts start    web.acme          # runs on spare

The master's flip reverses the msync data direction: the new master is authoritative, the old becomes a read-only replica. Identity (wireguard keys, IP, name) is preserved by default per FEAT-324; the cleanest way to carry wireguard explicitly is cpk hosts migrate ... --preserve-wg (FEAT-326 slice 10).

3.8. Tear down

cpk hosts down web.acme --remove-dns --release-ip
cpk machines destroy <machine>       # if no other host needs it

For ephemeral orgs (clones with --ephemeral=<n>):

cpk hosts destroy-org dev.example.com           # honour TTLs
cpk hosts destroy-org dev.example.com --force   # ignore TTLs
cpk hosts destroy-org staging.example.com.<ts> --after=14d
    # treat hosts cloned >= 14 days ago as expired

3.9. Observability

cpk hosts status   web.acme          # state + config/data freshness
cpk hosts monitor  web.acme          # tail logs across replicas
cpk cost --by host                   # spend rollup

3.10. Run a regression hook on a host or org

cpk hosts test web.staging.acme      # single host
cpk hosts test staging.acme          # every host in the org

Each cpk that ships .cpk/hooks/test has it exec'd locally with $HOSTNAME exported and cwd at the host repo (same shape as drain / resume). Hosts without the hook are reported skipped, not failed. Used by the cpk-upgrader pipeline (FEAT-326 slice 3).

4. Guardrails

1. Identity continuity is the default. Migrate / failover should preserve wireguard keys, mesh IP, and host name — downstream consumers (FEAT-320 deps) should not need to be told. Generate fresh identity only with an explicit flag.
2. hosts up <root> walks the dep graph. Don't bring up a leaf and assume its dependencies will exist; declare them in .cpk/config/depends and let up order them.
3. The provider boundary is a thin contract. When a verb has to branch on provider name in the operator path, that's a contract leak — surface it, don't paper over it. Cross-provider migration (FEAT-324) is the integration test for whether the contract holds.
4. Wire-format infra:* tags are stable. Cloud resources are tagged infra:managed/host/org (AWS) or infra/managed/host/org (Hetzner) for backward compat; do not retag silently. The user config path ${XDG_CONFIG_HOME}/infra/providers/ is also stable.
5. Drain before destruction. Any verb that stops serving traffic (stop, down, migrate) should pair with the edge's FEAT-318 503 + Retry-After for the in-flight window — cpk hosts split semantics, not a hard kill. The vendored edge cpk at share/cpk/edge/ ships drain/resume hooks that wire this into cpk hosts down --with-dependents automatically.
6. cpk container <verb>, not bare cpk <verb> for builder verbs. The dispatcher dispatches to container for build/run/depends/etc.; infra cpk build is now cpk container build. bin/infra ships as a one-release backward-compat shim.

5. Where to read more

• man cpk
• cpk hosts help <verb> for any verb above (every verb has a help block with its options + sequence).
• docs/cpk-walkthrough.md — author + operator walkthrough end-to-end
• docs/cpk.md — CLI contract reference
• docs/provider-driver.md — provider contract (what each driver implements)
• docs/roadmap.md — cloud-fleet roadmap (FEAT-300…309, 317-327 shipped)
• The companion cpk-author skill for authoring .cpk/ packages, and cpk-upgrader (FEAT-323 + FEAT-325/326/327) for the dev / staging / production upgrade pipeline.
• This package's CLAUDE.md.