| name | shipping-and-launch |
| description | Prepares production launches. Use when preparing to cut a release. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. |
Shipping and Launch
Overview
Ship with confidence. The goal is not just to release — it's to release safely, with monitoring in place, a rollback plan ready, and a clear understanding of what success looks like. Every launch should be reversible, observable, and incremental.
For this repo, "launch" means cutting a versioned release of the zynk binary: tagging a version, publishing a GitHub binary release, pushing the crate to crates.io, and updating the Homebrew tap. The same discipline applies whether you're shipping to package registries or rolling out a new agent-detection behavior to live runtimes.
When to Use
- Cutting a release of the binary to users for the first time
- Releasing a significant change to behavior (agent detection, IPC protocol, conversation layer)
- Migrating the on-disk DB schema or config format
- Opening a beta or early-access build
- Any release that carries risk (all of them)
The Pre-Launch Checklist
Code Quality
Security
Performance
Robustness
Infrastructure
Documentation
Feature Flag Strategy
Ship behind feature flags / config gates to decouple release from activation:
if config.flags.task_sharing {
return Some(render_task_sharing_panel(task));
}
None
Feature flag lifecycle:
1. RELEASE with flag OFF → Code is shipped but inactive
2. ENABLE for team/beta → Internal testing on real runtimes
3. GRADUAL ROLLOUT → opt-in flag → default-on for beta → default-on for all
4. MONITOR at each stage → Watch error rates, crash reports, user feedback
5. CLEAN UP → Remove flag and dead code path after full rollout
Rules:
- Every feature flag has an owner and an expiration date
- Clean up flags within 2 weeks of full rollout
- Don't nest feature flags (creates exponential combinations)
- Test both flag states (on and off) in CI
Staged Rollout
The Rollout Sequence
1. VALIDATE locally (isolated dev runtime)
└── Full test suite (`just check`) on an isolated CARGO_TARGET_DIR
└── Manual smoke test of critical flows against a dev socket/config (never the live one)
2. TAG + BUILD release artifacts (flag OFF)
└── Verify the release binary boots and reports the right version
└── Check the build is clean (no warnings, locked deps)
3. ENABLE for team (flag ON for internal/dogfood install)
└── Team runs the build as their live runtime
└── 24-hour monitoring window
4. CANARY rollout (publish to a beta channel / pre-release tag)
└── Monitor crash reports, error logs, regression reports
└── Compare: canary vs. previous stable
└── 24-48 hour monitoring window
└── Advance only if all thresholds pass (see table below)
5. PUBLISH widely (crates.io + GitHub release + Homebrew tap)
└── Same monitoring after publish
└── Ability to yank / point users back to the previous version
6. FULL rollout (default-on for all users)
└── Monitor for 1 week
└── Clean up feature flag
Rollout Decision Thresholds
Use these thresholds to decide whether to advance, hold, or roll back at each stage:
| Metric | Advance (green) | Hold and investigate (yellow) | Roll back (red) |
|---|
| Crash / panic rate | Within 10% of baseline | 10-100% above baseline | >2x baseline |
| Startup / hot-path latency | Within 20% of baseline | 20-50% above baseline | >50% above baseline |
| New error-log types | No new error types | New errors in <0.1% of sessions | New errors in >0.1% of sessions |
| User-facing regressions | None reported | Minor cosmetic only | Functional / data-loss reports |
When to Roll Back
Roll back immediately if:
- Crash/panic rate increases by more than 2x baseline
- Startup or hot-path latency increases by more than 50%
- User-reported issues spike
- Data integrity issues detected (corrupted DB, lost conversation state)
- Security vulnerability discovered
Monitoring and Observability
What to Monitor
Application signals:
├── Crash / panic rate (total and by code path)
├── Command latency (p50, p95, p99) for socket/IPC commands
├── Active panes / sessions
├── Error-log volume (by module via tracing targets)
└── Key behavior metrics (agent detection accuracy, message delivery)
Runtime / host signals:
├── CPU and memory utilization of the server process
├── PTY master FD count (leak detection)
├── Socket connection count
├── DB size growth
└── Background task / event-loop backlog
Client signals:
├── Render frame timing
├── Input-to-effect latency
└── User-reported error rates
Error Reporting
use tracing::error;
fn handle_command(cmd: Command, ctx: &Ctx) -> Result<Response, AppError> {
match dispatch(cmd, ctx) {
Ok(resp) => Ok(resp),
Err(err) => {
error!(
command = %cmd.name(),
session = ctx.session_id(),
error = %err,
"command failed"
);
Err(AppError::internal("something went wrong"))
}
}
}
The rules that matter:
- Never swallow an error silently — at minimum log it via
tracing with enough context to localize the failure.
- Don't expose internals to the user/client; map to a stable error shape.
- Attach context (command, session, target) so a single log line is enough to start debugging.
Post-Launch Verification
In the first hour after launch:
1. Check `zynk --version` reports the released version and the binary boots
2. Check error logs / crash reports (no new error types)
3. Check latency on the hot commands (no regression)
4. Test the critical user flow manually (spawn pane, run agent, exchange a message)
5. Verify logs are flowing and readable
6. Confirm rollback mechanism works (dry run: can you reinstall the previous version?)
Rollback Strategy
Every release needs a rollback plan before it happens:
## Rollback Plan for [Feature/Release]
### Trigger Conditions
- Crash/panic rate > 2x baseline
- Hot-path latency > [X]ms
- User reports of [specific issue]
### Rollback Steps
1. Disable feature flag / config gate (if applicable)
OR
1. Point users back to the previous version:
- crates.io: `cargo yank --version X.Y.Z` (then publish a fixed patch)
- GitHub release: mark the release as a draft / re-pin "latest" to the prior tag
- Homebrew tap: revert the formula to the previous bottle/version
2. Revert the code: `git revert <commit>` on a branch, then push — **push is an operator gate** (WORKFLOW.md rule 1); never force-push main
3. Verify rollback: `zynk --version`, error logs clean
4. Communicate: notify team of rollback
### DB / On-Disk Considerations
- Migration [X] is forward-only — document the manual downgrade path (drop index, delete the `_sqlx_migrations` row)
- Data written by the new feature: [preserved / cleaned up]
- Never assume tests isolate the live DB — verify migrations don't touch the live runtime's on-disk DB
### Time to Rollback
- Feature flag / config gate: < 1 minute
- Reinstall previous binary: < 5 minutes
- DB downgrade: < 15 minutes
See Also
- For security pre-launch checks, see
references/security-checklist.md
- For performance pre-launch checklist, see
references/performance-checklist.md
- For robustness/teardown verification before launch, see
references/robustness-checklist.md
Common Rationalizations
| Rationalization | Reality |
|---|
| "It works on my machine, it'll work for users" | Users have different terminals, agents, OSes, and edge cases. Monitor after release. |
| "We don't need a feature flag for this" | Every feature benefits from a kill switch. Even "simple" changes can break things. |
| "Monitoring is overhead" | Not having monitoring means you discover problems from user complaints instead of logs. |
| "We'll add monitoring later" | Add it before launch. You can't debug what you can't see. |
| "Rolling back is admitting failure" | Rolling back is responsible engineering. Shipping a broken release is the failure. |
| "Just publish to crates.io and tag it now" | Publish/tag/yank are separately gated, irreversible-ish operations. Validate locally first, then gate each one explicitly. |
Red Flags
- Releasing without a rollback plan
- No monitoring or error reporting on the live runtime
- Big-bang releases (everything at once, no staged validation)
- Feature flags with no expiration or owner
- No one monitoring the release for the first hour
- Release/version configuration done by memory, not by the lockfile and
Cargo.toml
- "It's Friday afternoon, let's tag and publish"
- Pushing, merging, publishing to crates.io, tagging, or yanking without a separate explicit operator gate
- Tests that migrate the live on-disk DB instead of an isolated one
Verification
Before releasing:
After releasing:
