test-service-shutdown-behavior

# test-service-shutdown-behavior Shutdown is a state transition, not just a signal handler. ## When to use - Use for servers, workers, schedulers, consumers, or stateful processes. - Never test destructive shutdown against production without authorized containment. ## Preconditions - Define signals, platform grace period, readiness and liveness semantics, traffic routing, task types, and durable state. ## Procedure 1. Record shutdown states and which new work each state accepts. 2. Verify readiness fails before termination while liveness remains appropriate. 3. Test connection drain, keepalive, streaming, and new-request rejection. 4. Test transactions, tasks, message acknowledgements, leases, locks, and side effects at each interruption window. 5. Enforce deadlines, cancellation, forced exit, and bounded cleanup. 6. Restart and reconcile incomplete work without duplication or loss. 7. Observe drain duration, abandoned work, errors, and platform kills. 8. Rehearse rolling deploy, scale-down, crash, and dependency delay. ## Failure plan - If graceful completion exceeds the platform deadline, make work checkpointable rather than extending shutdown indefinitely. ## Worked example A worker loses readiness, finishes a leased job, releases unused claims, exits before the deadline, and safely retries a pre-acknowledgement message after restart. ## Done - A shutdown test report records states, signals, drain, in-flight outcomes, deadlines, forced exit, restart, and reconciliation - Request, stream, task, transaction, lease, acknowledgement, rolling-deploy, and platform-kill tests verify behavior