| name | deployment-investigation |
| description | Use when investigating a failed, stuck, superseded, or ambiguous deployment across GitHub Actions, deployment event artifacts, mcl status summaries, target journals, cache health, and post-switch health checks. |
Deployment Investigation
Prerequisites
- Work from the repository that owns the deployment workflow and target list.
- Do not use deploy, cache, or target credentials until the failing target,
deployment id, and blast radius are known.
- Collect the GitHub run id, target name, git revision, deployment id, and event
artifact path if they exist.
Commands
gh run view "$RUN_ID" --log
gh run download "$RUN_ID" --dir .result/deployment-artifacts
mcl deploy-status summarize .result/deployment-artifacts/events.jsonl \
--output .result/deployment-summary.md \
--json-output .result/deployment-summary.json
ssh "$TARGET" 'sudo journalctl -u mcl-deploy-agent.service -u mcl-deployment-reconciler.service -b --no-pager -n 200'
ssh "$TARGET" 'sudo journalctl -u sshd.service -u ssh.service -b --no-pager -n 120'
nix path-info --store "$SUBSTITUTER" --recursive "$SYSTEM_PATH" \
--option trusted-public-keys "$TRUSTED_PUBLIC_KEY"
Use mcl deploy-status summarize first when an event JSONL artifact exists.
Use target journals next only for phases that reached the target:
agent-restore, switch, healthcheck, rollback, or complete.
Workflow
- Identify the final observed phase and
command.status from the status
summary.
- Join by
correlationId or deploymentId; do not mix events from separate
attempts.
- Classify the failure:
evaluate, build, or cache-push is controller-side;
activate-requested is transport or reconciler-side;
agent-restore, switch, healthcheck, rollback, and complete are
target-side.
- For cache suspicion, compare the manifest
desiredSystemPath with
nix path-info --store and the cache operation logs.
- For target suspicion, inspect the target journal and the desired-state status
files under
/var/lib/mcl/deployments.
Evidence
- GitHub run URL and failing job or step.
- Deployment summary markdown and JSON.
- Event JSONL lines for the failed deployment id.
- Target journal excerpt with timestamps matching the event stream.
- Cache query result for the exact system path and trusted public key.
- Health-check command, timeout, status, and output summary.
Stop And Ask
Stop and ask a human operator before changing desired state, retrying a
production target, rolling back, exposing secrets or tokens, rotating cache
keys, overriding host-key checking, or acting on a target whose name does not
match the signed manifest.
Rollback
Investigation does not roll back by itself. If rollback is required, hand off to
the break-glass workflow with the deployment id, target, current generation,
failed generation, and evidence collected above.
