cpk-upgrader

name

cpk-upgrader

description

Drive a fleet upgrade end-to-end: clone production to dev, apply the change, replay in staging against real data, run tests, get user sign-off, promote staging to production, with automatic disaster recovery on failure. Trigger when the user says "upgrade my nextcloud" / "bump postgres to 17" / "test this change against real data before deploying it" / any phrasing that maps to the dev → staging → production pipeline.

cpk-upgrader skill

Per FEAT-323 + FEAT-325/326/327. The third must-have role skill alongside cpk-author and cpk-operator: the upgrade pipeline the on-call / DevOps role drives. This skill describes the agent's workflow — the cpk verbs below are primitives, the agent is the operator that sequences them.

1. When to invoke

Trigger phrases that should pull in this skill:

• "upgrade my nextcloud instance"
• "bump postgres to 17 across the fleet"
• "test this change against real production data before deploying it"
• "I want to try X in staging with prod's data"
• "promote staging to production"
• "roll prod back to yesterday"
• "what's the safe way to upgrade "

Not this skill — out of scope:

• "create a new cpk" → cpk-author
• "deploy a host on a new machine" → cpk-operator
• "investigate why is down" → operator skill

2. The model

The agent treats environments as orgs, per the cpk convention (FEAT-326). For an example.com deployment:

Org	Purpose
example.com	production
staging.example.com	upgrade rehearsal with prod data
dev.example.com	author's sandbox (empty data)
preproduction.example.com	optional extra stage for risky bumps

The agent never invents secrets, never edits prod directly, never skips the staging step on a major upgrade.

3. The journey

3.1. Scope the upgrade

The user says "upgrade nextcloud". The agent:

• Asks what version / what change (or reads the changelog if the user passes a URL).

• Identifies the impacted host(s) by walking the dep graph:

  cpk container nextcloud.example.com depends --transitive
  cpk hosts graph nextcloud.example.com           # human-readable
  

• Names the upgrade plan back to the user before doing anything destructive.

3.2. Try in dev (fast, empty data)

cpk hosts clone-org example.com dev.example.com \
    --machine-type cx22=cx11 --ephemeral=7d

--empty-data is the default; the cloned hosts come up empty. The --machine-type mapping is honoured by cpk hosts up as the default --type when provisioning. --ephemeral=7d writes an expiry timestamp so a later cpk hosts destroy-org reaps the short-lived org automatically.

Then the agent edits the .cpk/ (e.g. bumps a package version, adds a Dockerfile fragment), commits, and brings it up:

cd nextcloud.service
# … edits to .cpk/<fragment> per the user's intent …
git commit -am "nextcloud: bump to 28.0.4 (FEAT-WHATEVER)"

cpk hosts up nextcloud.dev.example.com

Smoke check: does it boot? Does the .cpk/check pass? Does the service respond?

3.3. Replay in staging (slow, real data)

cpk hosts clone-org example.com staging.example.com --with-data

--with-data records the source-host pairing under .cpk/config/cloned-from for each new host. Then per host (after cpk hosts up provisions the target machines):

cpk hosts seed-data nextcloud.staging.example.com \
    --handler=file,postgres

This streams the source snapshot tree source → target over an ssh-to-ssh pipe (no orchestrator staging). For schema-only clones (test migrations without copying gigabytes of rows), add --schema-only to clone-org; postgres backup will then dump schemas only.

If the user opts in to --inherit-secrets on the clone, follow up with:

cpk hosts seed-secrets nextcloud.staging.example.com

This reads .cpk/config/secrets (one secret name per line) and copies each via ssh src 'secret get N' | ssh tgt 'secret set N -' — values never land on the orchestrator's disk.

Then the agent replays the same change-set in staging (cherry-pick the dev commits, or re-edit):

cd nextcloud.service
git checkout dev-commit-or-branch
cpk hosts up nextcloud.staging.example.com

This is where real data meets the upgrade. Common failures that only show up here: schema migrations on a real-sized database, file-format changes, plugin compatibility.

3.4. Run the regression suite

cpk hosts test staging.example.com    # runs each cpk's .cpk/hooks/test

If the cpk ships its own smoke tests (a .cpk/hooks/test hook), cpk hosts test runs them. The agent surfaces failures verbatim to the user. Skipped hosts (no hook shipped) count as "silent OK", not as failures.

For the inspection step BEFORE promote, two diff verbs help the operator see what's about to change:

cpk hosts snapshot-diff nextcloud.staging.example.com \
    nextcloud.example.com                                # file data
cpk hosts snapshot-diff nextcloud.staging.example.com \
    nextcloud.example.com --handler=postgres             # DDL changes

3.5. Get explicit user sign-off

Show the user:

• What changed in staging vs prod: git log on the cpk + cpk hosts snapshot-diff for the live data + DDL deltas.
• What the regression suite said.
• A link / screenshot of staging if possible.
• The rollback anchor that will be created in step 3.6.

Ask, don't assume. The agent does not promote without an explicit "go". This is the moment a human catches the thing the regression suite missed.

3.6. Anchor + promote

cpk hosts snapshot nextcloud.example.com file backup   # per host
cpk hosts promote-org staging.example.com example.com --swap --dns-flip

cpk hosts snapshot <host> file backup writes a timestamped backup on the host's master; that timestamp becomes the rollback id for step 3.7. Per-host retention via .cpk/config/retention (an integer) is pushed to the remote's max-backups automatically before the backup runs.

promote-org --swap --dns-flip does the atomic blue/green: archives the existing prod org to <svc>.<target-org>.<ts>, renames staging to prod, and re-upserts A records at the promoted machines. The drain-window UX is provided by the edge cpk's drain hook (FEAT-318 slice 6 -- serves 503 + Retry-After during the cut-over).

For an extra safety belt: cpk hosts snapshot-export <host> --pass-file <p> streams the pre-promote snapshot tree to an encrypted offsite location (openssl aes-256-cbc) for the worst case where the in-cluster rollback also fails.

3.7. Disaster recovery (when things go wrong)

If post-promote monitoring shows issues — broken endpoint, errors in the logs, user reports — the agent rolls back:

cpk hosts snapshot nextcloud.example.com list      # find the pre-upgrade id
cpk hosts rollback nextcloud.example.com --to <id>

The id is pre-validated against snapshot list BEFORE the drain runs — a typo in the id won't take the host down. Without --to, rollback uses the snapshot tool's latest.

The previously-active staging org is preserved as staging.example.com.<ts> for forensic analysis.

For image-only providers (fly.io), data lives on a volume rather than the in-VM snapshot tool. Use the provider-native verb:

cpk hosts volume-snapshot nextcloud.example.com list
cpk hosts volume-snapshot nextcloud.example.com restore <snapshot-id>

3.8. Archive

cpk hosts destroy-org staging.example.com.<ts> --after=14d

Keep the swap-archived prod org for two weeks in case a delayed regression surfaces. --after=14d treats every host whose .cpk/config/ephemeral is older than 14 days as expired and tears it down; hosts inside the window are skipped. Without --after, the verb honours each host's own .cpk/config/ephemeral expiry.

4. Guardrails

1. Never skip staging on a major upgrade. Major = changes the data format, the API surface, or any cross-host contract. The agent asks "is this a major upgrade?" and refuses --upgrade mode (in-place) when the answer is yes.
2. Always anchor before promote. A cpk hosts snapshot before every promote-org --swap. No exceptions.
3. Tests run on real data in staging, never in production. This is the whole point of the pipeline. If staging fails, prod never moves.
4. The agent does not invent secrets. Production secrets never flow into staging unless --inherit-secrets is explicitly passed by the user.
5. Drain hooks are the cpk author's contract. The agent does not bypass them. If a cpk's .cpk/hooks/drain exits non-zero, the orchestrator aborts; the agent surfaces the error rather than --ignore-drain-failures past it.
6. The user signs off on promote. Always. Even when the regression suite is green. The agent surfaces the diff + tests and asks before swapping.
7. Rollback is a first-class step, not a last resort. The agent runs it cheerfully when the user reports an issue — the snapshot anchor exists to be used.
8. cross-org=no is hard-coded into the model. Each org is its own world; no dep on a different org's host. The agent does not propose otherwise.

5. The primitives this skill consumes

Dep-graph + per-host orchestration (FEAT-320 / 324 / 325):

Need	Verb
Walk dep graph for scope	cpk container depends --transitive
Visualise dep graph	cpk hosts graph
Derive host zone	cpk hosts zone <host> (FEAT-317)
Drain a dependent cleanly	cpk hosts drain + .cpk/hooks/drain
Resume a dependent	cpk hosts resume + .cpk/hooks/resume
Bring up a closure	cpk hosts up <root> --with-dependents
Tear down a closure	cpk hosts down <root> --with-dependents
Move single host	cpk hosts migrate
Multi-host migrate bracket	cpk hosts migrate --with-dependents
Carry wireguard identity	cpk hosts migrate --preserve-wg

Org lifecycle (FEAT-326):

Need	Verb
Clone an org	cpk hosts clone-org <src> <tgt>
Carry source-host pairings	clone-org --with-data
Carry secret intent	clone-org --inherit-secrets
Carry instance-size hint	clone-org --machine-type <t|a=b>
Schedule auto-destroy	clone-org --ephemeral=<n>[d]
Schema-only postgres dumps	clone-org --schema-only
Seed file + postgres data	cpk hosts seed-data <host> --handler=file,postgres
Seed declared secrets	cpk hosts seed-secrets <host>
Run regression	cpk hosts test <host-or-org> + .cpk/hooks/test
Promote an org (blue/green)	promote-org --swap --dns-flip
Tear down by TTL	cpk hosts destroy-org <org> [--force|--after=<n>[d]]

Snapshot + recovery (FEAT-327):

Need	Verb
Snapshot anchor	cpk hosts snapshot <host> file backup
Per-host retention	.cpk/config/retention integer file
Snapshot history	cpk hosts snapshot <host> list
Disaster rollback (latest)	cpk hosts rollback <host>
Named-snapshot rollback	cpk hosts rollback <host> --to <id>
Inspect file diff	cpk hosts snapshot-diff <a> <b>
Inspect DDL diff	cpk hosts snapshot-diff <a> <b> --handler=postgres
Encrypted offsite handoff	cpk hosts snapshot-export <host> --pass-file <p>
Image-only provider snapshots	cpk hosts volume-snapshot <host> <create|list|restore>

Edge (FEAT-318):

Need	Verb
Edge route table	cpk hosts edge <root>...
Maintenance window	cpk hosts edge ... --mode=maintenance
Per-root maintenance	cpk hosts edge ... --maintenance=<root>,...
Wildcard certs	cpk hosts edge ... --acme=dns01:<provider>
Per-route auth	.cpk/config/auth (path / user / bcrypt rules)
L4 (TCP/UDP) forward	.cpk/config/l4 (port[/proto] upstream rules)
Deployable edge container	share/cpk/edge/ (copy to <repo>/edge.service)

All primitives above are shipped. The upgrade pipeline is fully usable end-to-end.

6. Where to read more

• man cpk
• cpk hosts help <verb> for any verb above (every primitive has a help block with its options + sequence).
• docs/cpk-walkthrough.md — author + operator walkthrough.
• share/doc/cpk/standards/ — the format spec
• The sibling cpk-author skill (creating a cpk).
• The sibling cpk-operator skill (running a fleet).