test-then-release

name

test-then-release

description

Staged canary ladder for taking a freshly-cut Agent Bridge version from tag to full-fleet deploy — VM (connected + disconnected tracks) → dev farm → prod farm → keystone (operator's own box) — verifying per-rung acceptance and iterating (fix → re-cut → re-test) on any failure, then reporting. Use whenever the operator says "테스트해서 릴리즈 해줘" / "test then release" / "roll this cut out to the fleet" / "canary this version", or after any release cut when the version must be validated on real environments before fleet-wide adoption. Encodes the fixed rung order, per-rung acceptance sets, remote-reach mechanics (A2A to each server's own patch — never operator ssh), the autonomy vs operator-gate boundary (prod visibility report, cut/tag/release, keystone confirm-first, unrecoverable-access HOLD), and the hard-won gotchas from the 2026-07-08 v0.17.4 rollout (macOS setsid absence, attached-agent restart-required, self-restart-forbidden, connected-gated fires only when updater-connected, Touch-ID ssh gate).

Test-then-release — staged canary ladder (VM → dev farm → prod farm → keystone)

When to use this skill

Use when:

• The operator says "테스트해서 릴리즈 해줘", "test then release", "canary this version", "roll it out to the servers", "fleet deploy vX.Y.Z".
• A version has just been cut/tagged and must be validated on real environments before fleet-wide adoption.
• A durable fix (token rotation, daemon behavior, auth) needs connected-path live verification beyond smoke + code review — the smoke/code layer proves the logic; the ladder proves it in a real registered environment.

Do NOT use when:

• The change is code-review/smoke-level only and nothing about it depends on a live registered environment (a docs PR, a pure CLI flag). Verify at the PR gate, no ladder needed.
• The operator asks for a single-host manual apply (agb upgrade on one box) with no fleet intent — just do that host safely (see §Safety).

Cut vs deploy boundary: this skill starts after a tag exists. Cutting the tag / GH release itself is a separate, operator-gated step — cut/tag/release are irreversible + outward-facing and always need an explicit, per-release operator go. A blanket "test then release" or standing ladder autonomy does not authorize the cut itself (see CLAUDE.md "Releases require explicit operator permission"; docs/release-lines.md; the wave-orchestration Phase-5 rule that standing autonomy never extends to the release ship). Never cut inside this skill — the tag must already exist when you start.

The fixed ladder (order is invariant)

Each rung: install → verify acceptance → "no big problem" → next rung. Order never changes — dev before prod, keystone last.

Rung 1 — VM canary (isolated sandbox, host live untouched)
  ├─ Track A: CONNECTED   — token-updater-registered VM → proves authority/rotation actually fires
  └─ Track B: DISCONNECTED — unregistered VM → proves byte-identical / safe-fallback invariant
        (run both; Track B de-risks the upgrade mechanism for Track A)

Rung 2 — dev farm
  └─ upgrade + run + verify. Dev canary MUST precede prod — never skip.

Rung 3 — prod farm  ⚠ PRODUCTION
  └─ send the operator a one-line status BEFORE crossing into prod (visibility, not a gate,
     when the operator has pre-authorized the whole ladder) — then proceed.

Rung 4 — keystone (operator's own box)  ← LAST
  └─ local `agb upgrade`. Highest-blast-radius (operator channel + your own session
     live here). Restart that touches the operator channel is confirm-first.

Why keystone last: it hosts the operator's channel and your own admin session. A wedge here is the most visible and (for your own session) the least self-recoverable failure — you burn down the canary risk on everything else first.

Per-rung acceptance set

Baseline acceptance for every rung:

• VERSION == target (agent-bridge upgrade --check --ref vX.Y.Z → installed == target, update_available=no)
• daemon health=ok, daemon_liveness=up, tick_fresh=true (clean SIGTERM re-start on upgrade, no crash-loop)
• no wedge / no stuck picker; agents survive the restart
• no new regression (diff any test failures against the pre-upgrade baseline — pre-existing failures ≠ regressions)

Feature-specific acceptance depends on connected vs disconnected (this was the crux of the v0.17.4 authority-follow fix — it only fires when updater-connected):

• Connected (Track A / registered farms): updater lease --check rc=0, last-token-lease-ok fresh (age < ~660s) with fail<2, the gated feature armed (e.g. BRIDGE_UPDATER_AUTHORITY_ENABLED=1 in the live daemon env + authority_mode=1 when connected), and if observable, the feature actually fires (central swap executes, not local fallback). If a live fire is hard to observe, run the shipped smoke against an isolated temp BRIDGE_HOME (never the live daemon) as fire-level evidence, and note it as smoke-not-live.
• Disconnected (Track B / unregistered hosts): connected=false → feature is no-op / byte-identical to a dark install → verify install + daemon up + VERSION + zero regression. This is the safe-fallback invariant: default-ON is fail-safe precisely because unconnected hosts no-op.

Record which rungs got live-fire evidence vs smoke-only — say so honestly in the report.

Reach mechanics — how you touch each environment

Rung	Env	How you reach it
1	local VM (OrbStack Linux / tart macOS)	you drive directly — isolated sandbox, host live state untouched
2–3	remote farm	A2A to that server's own patch agent — agent-bridge a2a send --peer <peer> --to patch … then a2a deliver. That patch runs agent-bridge upgrade --apply --ref vX.Y.Z with local privileges.
4	keystone (this box)	local agb upgrade (heavy → bg + long timeout + monitor + fallback)

★ Never operator-ssh into a farm. Every Agent Bridge server runs its own local patch. Remote management goes through that patch via A2A, not your ssh.

• known_agents=- in the peer list is NOT ground truth — it's your local A2A registration cache only. A peer showing - still receives a2a send --to patch and acks with a remote task id (a peer has shown - yet acked a remote task and self-upgraded). Attempt-first; don't pre-declare "no agent there."
• Farm ssh is typically behind an interactive signing gate (e.g. a Touch-ID / hardware-key SSH agent) anyway — it hangs on the final signature unless the operator physically approves. Background bash can't reach the interactive signing agent. So ssh is both wrong (bypasses the farm's own patch) and usually impossible. If a rung genuinely has no reachable patch and no A2A, it's an operator-unlock HOLD, not a "do it via ssh."

The iterate loop

Any rung failure (regression, wedge, acceptance miss) → stop the ladder → fix → cut a new version (the cut itself is operator-gated — see the Autonomy boundary) → re-test from Rung 1. A new cut is a new artifact, so it restarts the whole ladder (Rung 1 VM Track A + Track B → dev → prod → keystone) — never resume a new version at the rung where the old one failed. Counter-example this forbids: v1 passes VM+dev but fails at prod; a fix makes v2; resuming v2 "at the affected rung" would push a never-canaried v2 straight to prod, skipping VM+dev — the exact "dev before prod — never skip" violation. The only case that may resume at the affected rung is a retry with no artifact change (e.g. a transient infra flake, re-running the same version). Each fix gets its own version; never patch a live host out-of-band and call it released. If the fix needs a source change it goes through the normal codex-pair review + agb-dev cut path — the ladder doesn't authorize skipping review for speed.

Autonomy vs operator-gate boundary

Autonomous (when the operator has pre-authorized the ladder / granted off-hours autonomy):

• Rung 1–2 install + verify + iterate.
• Rung 3 prod — proceed only if all three hold: (a) Rung 2 dev passed, (b) recovery access to the prod host is held (a reachable farm patch or an ssh path you can actually complete — not a gate you can't approve at night), and (c) the operator has pre-authorized the prod crossing. Send a one-line status before crossing (visibility). If recovery access is not held → HOLD, do not cross on visibility alone.
• On a rung failure, autonomously fix + prepare the next version (branch, draft VERSION/CHANGELOG, run pair-review) on non-prod rungs. But the cut itself is not autonomous — it HOLDS for an explicit per-release operator go (next item). Testing resumes only once the operator approves the cut and the new tag exists, and it resumes from Rung 1 (a new artifact re-walks the whole ladder — see The iterate loop), not from the failed rung.

Confirm-first / operator-gated — never autonomous even under a ladder grant:

• Cut / tag / GH release (irreversible + outward-facing) — always requires an explicit, per-release operator go. Standing ladder autonomy (or a blanket "test then release") never extends to the release ship itself (CLAUDE.md "Releases require explicit operator permission"; docs/release-lines.md; wave-orchestration Phase 5). This skill assumes the tag already exists.
• Prod-farm authority-enable / any prod deploy where recovery access is not held — if a prod upgrade could wedge the daemon and you'd have no way back in (an interactive-signing ssh you can't approve at night, no local farm patch), that rung is HOLD until the operator unlocks recovery access. Unrecoverable prod mutation at night is the bright line — code-only + authority=0 still restarts the daemon and still mutates prod, so it carries the same recovery risk.
• Keystone restart that touches the operator channel — confirm-first.
• Your own (the admin agent's) restart — you cannot cleanly self-restart (a session can't kill itself mid-command and resume). Delegate to your codex pair (<admin>-dev): queue a task, it runs agent-bridge agent restart <admin>, verifies return, recovers on failure, and re-queues a resume task; leave a NEXT-SESSION.md backup.

If you can't proceed and can't safely gate-cross, surface blocked with the real reason — don't silently stall, and don't fabricate an "it's impossible" from a cache field.

Safety discipline (destructive live ops)

From the #2085 destructive-ops incident:

• Heavy ops (agb upgrade, daemon restart) = background + long timeout + log monitor + fallback. Never a single foreground call — upgrade takes 2min+ (plugin installs); a default-timeout SIGKILL mid-upgrade leaves a partial apply / version skew.
• One destructive change at a time, verify between each.
• Pre-verify the environment matches your assumptions before a destructive step (e.g. is the daemon launchd-managed or plain-bash? launchctl print …).
• Daemon down → bash bridge-daemon.sh start first (UP is priority 1), then re-arm launchd supervision (priority 2).
• Daemon supervision is the highest-risk surface — no improvised repetition; lean on the domain owner (agb-dev).

macOS detach gotcha

A keystone/self-affecting upgrade can be killed if your own session dies mid-run (the upgrade restarts the daemon, which can take your agent process down with it) → partial apply / version skew. It must be session-detached + backgrounded + logged, then monitored, with a fallback.

macOS has no setsid(1) binary, so use Python's os.setsid() to move the upgrade into its own session (immune to the parent session's death). This is a single setsid, not a double-fork — describe it as session-detach, nothing more:

LOG=~/.agent-bridge/state/upgrade-vX.Y.Z.log
python3 -c "import os,subprocess,sys; os.setsid(); subprocess.run(sys.argv[1:])" \
  bash -lc "agent-bridge upgrade --apply --ref vX.Y.Z >$LOG 2>&1" &
echo "upgrade pid=$!"          # record the PID

Then: monitor $LOG for completion; if it stalls or the daemon goes down mid-upgrade, fallback — bash ~/.agent-bridge/bridge-daemon.sh start first (UP is priority 1), then re-arm launchd supervision (priority 2), then reconcile the version. Never fire a keystone upgrade as a bare foreground call.

Fleet-restart skips attached agents

bridge-upgrade's fleet-restart skips any agent with a tmux client attached → that agent stays on the old version with a lingering restart-required task. Follow up per-agent. For the admin agent itself, that follow-up is the codex-pair-driven restart above (self-restart forbidden).

Known upgrade robustness gap (track upstream)

bridge-upgrade runs the daemon restart before fleet-restart; if the session is SIGKILLed in that window the fleet-restart + its work-complete marker can be lost. Durable fix = daemon-resume that consumes the work-complete marker at daemon startup (idempotent + locked + fail-open) — agb-dev GH #2356 class. Watch for it; adversarially review when it lands.

Reporting shape

At ladder completion, report to the operator in plain language (no internal jargon — plain words, not authority_mode/silent non-delivery):

• what shipped, and the rung order actually walked
• per-rung: pass/fail, and connected vs disconnected coverage (which rungs got live-fire vs smoke-only evidence)
• anything held / gated and why (with the specific operator decision needed)
• version / daemon health / feature-armed confirmation

Farm status that the operator cares about is reported by that farm's own patch, directly — don't relay-duplicate it through yourself.

One-glance checklist

1. Tag exists? (cutting the tag is a separate step needing an explicit per-release operator go — never cut here on standing autonomy)
2. Rung 1 VM: Track A connected (authority fires) + Track B disconnected (byte-identical). Both pass?
3. Rung 2 dev farm via its patch (A2A, not ssh). Pass?
4. Rung 3 prod farm: one-line status to operator → cross → verify. Recovery access held? If not → HOLD.
5. Rung 4 keystone local upgrade (bg + long timeout + fallback + macOS python-detach). Operator-channel restart = confirm-first. Admin self-restart = delegate to your codex pair (<admin>-dev).
6. Any failure → fix → re-cut (operator-gated) → re-test from Rung 1 — a new artifact re-walks the whole ladder; only a no-artifact-change retry (same version) may resume at the affected rung.
7. Report: rungs walked, connected/disconnected coverage, live-fire vs smoke, holds + reasons.