pilotswarm-aks-reset

## Select the execution path Read `.github/DEPLOYMENT.md` first. The commands below are operator-managed environment recipes. For this repository’s managed environment, local Azure or Kubernetes mutations are not permitted. Identity setup is a one-time admin prerequisite; ongoing changes/reset/teardown need an authorized maintenance Action, which is not currently supplied. Do not run these recipes locally as a fallback. Preserve the selected auth posture; VPN steps apply only to a VPN-enabled target. # PilotSwarm AKS Reset Use this skill when the user explicitly asks to wipe remote PilotSwarm state, reset AKS-backed databases, purge blob-backed session state, or recover from tainted orchestration history. This is a destructive workflow. Be exact about what will be lost and use the repo-owned reset path. **NO RESETS unless the user explicitly asks.** Resets are never part of a deploy: `scripts/deploy-aks.sh` never touches data, and there is no reset flag in it. The only reset entry point is `scripts/reset-db-aks.sh`, which requires the literal flag `--i-understand-this-deletes-all-data`. "Deploy", "redeploy", "clean rollout", or an orchestration-version change is NOT a request for a reset — if a reset seems warranted, say so and wait for the user to explicitly ask for a wipe. ## What The Reset Destroys The canonical reset script drops the PilotSwarm schemas and may also purge blob-backed dehydrated session state: - `duroxide` or `DUROXIDE_SCHEMA` - `copilot_sessions` or `CMS_SCHEMA` - `pilotswarm_facts` or `FACTS_SCHEMA` - all blobs in the configured Azure container when `AZURE_STORAGE_CONNECTION_STRING` is present That means: - all in-flight orchestrations are lost - all session rows/events are lost - all durable facts are lost - all dehydrated session archives/checkpoints in blob storage are lost ## Canonical Files - Reset entry point: `scripts/reset-db-aks.sh` (requires `--i-understand-this-deletes-all-data`; handles scale-down → wipe → scale-up) - Underlying reset script: `scripts/db-reset.js` (invoked by the wrapper) - Deploy script: `scripts/deploy-aks.sh` (never resets) - Orchestration debugging skill/context: `.github/skills/debug-orchestration/SKILL.md` - AKS guide: `docs/developer/deploy/aks.md` ## When Reset Is The Right Call - orchestration code changed and the user wants a clean AKS rollout - stale history is causing replay or nondeterminism problems - old pods from another namespace polluted the shared database - the user explicitly wants to wipe remote sessions/facts before testing again ## Canonical Reset Workflow 1. Confirm the target. - For AKS, use `.env.remote` if present. - Be explicit that this is the remote environment, not the local test database. 2. Stop workers before wiping state. - Scale the deployment to zero so nothing repopulates state during reset: ```bash kubectl scale deployment copilot-runtime-worker -n copilot-runtime --replicas=0 ``` 3. Check for stray workers in other namespaces if the reset is related to nondeterminism. - Run: ```bash kubectl get pods --all-namespaces -l app.kubernetes.io/component=worker --no-headers ``` - Delete unexpected old deployments before reintroducing new workers. 4. Run the canonical reset command. - Follow the deploy script's remote-safe pattern: ```bash NODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts/db-reset.js --yes ``` - This handles Azure/Postgres SSL quirks and also purges blobs when blob storage is configured. 5. Redeploy or restart workers after the reset. - If new code or secrets need to ship, build and push the image while workers are still at zero replicas. - Apply manifest changes if any: `kubectl apply -f deploy/k8s/worker-deployment.yaml` - Scale back up: `kubectl scale deployment copilot-runtime-worker -n copilot-runtime --replicas=6` - **Do not use `kubectl rollout restart`** — it does a rolling replacement that mixes old and new pods, which can cause session-state-not-found failures when in-flight orchestrations cross the old/new boundary. - Wait for rollout completion: `kubectl rollout status deployment/copilot-runtime-worker -n copilot-runtime --timeout=180s` - Validate worker filesystem ownership before starting tests: ```bash for p in $(kubectl get pods -n copilot-runtime -l app.kubernetes.io/component=worker -o jsonpath='{.items[*].metadata.name}'); do kubectl exec -n copilot-runtime "$p" -- sh -lc 'touch /home/node/.copilot/config.json >/dev/null 2>&1 && echo ok || echo broken' done ``` 6. Verify the cluster is truly clean. - Pods should come back healthy. - The session list should not show stale pre-reset user sessions. - Expect the built-in system sessions to be recreated automatically as the workers boot. - If the reset was to remove a provider, verify that the live model surface also changed after restart. - If the reset was to reproduce a system-agent failure, verify whether the recreated root `PilotSwarm Agent` is healthy before starting new test sessions. ## Preferred Commands - Canonical reset (scale-down → wipe → scale-up, one command): ```bash ./scripts/reset-db-aks.sh --i-understand-this-deletes-all-data ``` Note: `./scripts/deploy-aks.sh` no longer resets anything — a "destructive redeploy" is this reset followed by a normal deploy. - Reset with image update (the safe order matters): ```bash # 1. Scale workers to zero — no pods should be running during reset kubectl scale deployment copilot-runtime-worker -n copilot-runtime --replicas=0 kubectl rollout status deployment/copilot-runtime-worker -n copilot-runtime --timeout=60s # 2. Wipe remote state (DB schemas + blob storage) NODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts/db-reset.js --yes # 3. Build and push new image (if code changed) az acr login --name "${ACR_NAME:?set ACR_NAME in .env.remote}" # NPM_REGISTRY is required on managed devices (public npm is blocked, and the # container inherits the block); .env.remote sets it. Harmless elsewhere. docker buildx build --platform linux/amd64 -f deploy/Dockerfile.worker \ --build-arg NPM_REGISTRY="${NPM_REGISTRY:-https://registry.npmjs.org/}" \ -t "${ACR_NAME:?set ACR_NAME in .env.remote}.azurecr.io/copilot-runtime-worker:latest" --push . # 4. Apply any manifest changes, then scale back up kubectl apply -f deploy/k8s/worker-deployment.yaml kubectl scale deployment copilot-runtime-worker -n copilot-runtime --replicas=6 kubectl rollout status deployment/copilot-runtime-worker -n copilot-runtime --timeout=180s ``` - Reset only (no image change, same code): ```bash kubectl scale deployment copilot-runtime-worker -n copilot-runtime --replicas=0 kubectl rollout status deployment/copilot-runtime-worker -n copilot-runtime --timeout=60s NODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts/db-reset.js --yes kubectl scale deployment copilot-runtime-worker -n copilot-runtime --replicas=6 kubectl rollout status deployment/copilot-runtime-worker -n copilot-runtime --timeout=180s ``` ## Learnings To Preserve - Reset first when orchestration behavior changed in a way that could alter replay history. - Blob purge is part of the reset story. If blob storage is configured, wiping only PostgreSQL is not a full reset. - Namespace drift is a real failure mode. Old workers in an old namespace can keep polling the same database and poison new deploys. - Scaling workers to zero before reset is not optional if you want a clean wipe. - **Never use `rollout restart` as a substitute for scale-down/scale-up.** A rolling restart replaces pods one at a time, meaning old pods with stale in-memory session state coexist with new pods pulling from a freshly-wiped database. This causes "expected resumable Copilot session state ... but none was found" failures when a session's `continueAsNew` execution lands on a new pod that has no local disk state from the old pod's checkpoint. Always: scale to 0 → reset DB → (push image if needed) → scale back up. - An empty post-reset catalog is temporary. Healthy workers will immediately recreate the built-in system sessions. - If local `kubectl` auth is unreliable, verify or operate through a cluster-side path instead of guessing whether the reset worked. - If fresh sessions fail right after reset with `CLI server exited with code 1` or `Connection is closed`, check whether `/home/node/.copilot` is writable inside each worker pod. Mounting only `/home/node/.copilot/session-state` is not sufficient for the Copilot CLI. ## Rules - Never run a remote reset without explicit user permission. - Never describe a reset as "just the DB" when it will also wipe facts and blob-backed session state. - Never redeploy after a reset without verifying the worker rollout completed.