macos-incident-postmortem

# macOS Incident Postmortem ## When to Use This Skill | Use this skill when... | Use something else when... | |------------------------|----------------------------| | GUI froze and you're not sure if the machine rebooted | Live-debugging a hung process — use `sample` / `spindump` | | Investigating recent kernel panics or watchdog timeouts | Application crashes — open the per-app `.crash` / `.ips` directly | | Cross-referencing "what was I doing at time T?" against logs | Active CPU diagnosis — see `launchservices-health` | | Auditing whether the system is "due for a reboot" | Pre-incident hardening — wrong skill, this is forensics | ## Platform Guard This skill is **macOS-only**. `/Library/Logs/DiagnosticReports/`, `kern.boottime`, `pmset -g log`, and the `log` command are Darwin-specific. Refuse to act if `uname -s` is not `Darwin`. ```bash test "$(uname -s)" = "Darwin" || { echo "macos-plugin: not Darwin, refusing"; exit 1; } ``` ## Core Expertise A macOS "incident" can mean any of: - A kernel panic (system reset by the kernel) - A WindowServer userspace watchdog timeout (the GUI froze and the user power-cycled) - A LaunchServices / coreaudiod / mds-stores XPC stall (one daemon dragged the GUI down) - A jetsam memory-pressure event (the kernel killed apps to reclaim RAM) - A thermal throttle (CPU clamped to base frequency for minutes) - A user-initiated force-reboot after a hang (no panic; just lost state) The first job in a postmortem is **distinguishing between actual reboots and GUI hangs**. The 2026-04-22 incident that motivated this plugin *looked* like a crash to the user but `last reboot` showed no reboot — the kernel was fine, only the GUI stack hung. The second job is **timeline reconstruction**: cross-reference Diagnostic Reports, `kern.boottime`, `last reboot`, `last shutdown`, and shell history to answer "what happened around time T?". ## Gather the deterministic signals Run the bundled script to gather every mechanical signal in one pass — reboot-vs-hang detection, per-category DiagnosticReports counts, the CPU-event offender histogram, the jetsam victim list, and the fixed reboot/hang classifier: ```bash bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh" --home-dir "$HOME" ``` Parse `STATUS=` and `ISSUES:` from the output. The script emits `CLASSIFICATION=` (REBOOT_PANIC / REBOOT_CLEAN / HANG_UI / HANG_CPU / HANG_JETSAM / HANG_OR_POWERCYCLE / UNKNOWN) with a `CLASSIFICATION_REASON=`, plus `PANIC_COUNT=` / `HANG_COUNT=` / `JETSAM_COUNT=` / `CPU_RESOURCE_COUNT=` etc., `LATEST_PANIC=` / `LATEST_HANG=` paths, and `CPU_OFFENDERS:` / `JETSAM_VICTIMS:` histograms. Pass the incident time T as an epoch to enable the classifier (`MACOS_PM_INCIDENT_EPOCH=<epoch> bash …`), or supply an alternate report source with `--reports-dir <path>`. With no incident time the classifier reports `UNKNOWN` rather than guessing — feed it T once you know it. The reboot/hang decision tree the script encodes: | Pattern | Interpretation | |---------|----------------| | Boot at/after T with a `.panic` report | True reboot from a kernel panic | | Boot at/after T, no panic | Clean restart (user-initiated or watchdog) | | Boot before T with a `.hang`/`.spindump.txt` | GUI hang — machine did not reboot | | Boot before T with a `.cpu_resource.diag` | Daemon CPU storm | | Boot before T with a `JetsamEvent-*` | Memory-pressure kill | | Boot before T, none of the above | Power loss or hard power-cycle | When a report filename or category is unfamiliar, or `last reboot` comes back empty because `wtmp` rotated past T, open [references/diagnostic-reports.md](references/diagnostic-reports.md) for the suffix → category → severity table and the unified-log boot/shutdown query. ## Timeline Reconstruction (judgment) With the deterministic classification and counts in hand, the remaining steps are the agent's job: read the panic backtrace, name suspect kexts, correlate with logs and shell activity, and write the narrative. ### Step 1: Inspect the unified log around T ```bash # Adjust the time window to bracket T log show --start "2026-04-22 08:15:00" --end "2026-04-22 08:25:00" \ --predicate 'subsystem == "com.apple.WindowServer" OR process == "launchservicesd" OR process == "coreaudiod"' \ --style syslog \ | head -500 ``` Common signatures to grep for: | Signature | Meaning | |-----------|---------| | `WindowServer:` watchdog | UI froze long enough to trip the watchdog | | `posix_spawn` failures | Fork/exec storm — usually shell loops or runaway scripts | | `Jetsam Killing` | Kernel killing processes for memory | | `_dispatch_*_timeout` | Daemon stuck on a synchronous IPC call | | `Thermal pressure` | CPU thermal-throttled | ### Step 2: Inspect the panic / hang report The script's `LATEST_PANIC=` / `LATEST_HANG=` lines give you the file to read. Key fields in a panic report: | Field | Meaning | |-------|---------| | `panic(cpu N caller ...)` | The instruction that panicked; first line says why | | `Backtrace (CPU N)` | Call stack at the time of panic | | `Mac OS version`, `Kernel version` | OS state | | `System uptime in nanoseconds` | Uptime at the moment of panic | | `last loaded kext` / `loaded kexts` | Likely third-party suspect | Hang reports are spindump-style: one column per thread of the hung process (typically WindowServer or the offender daemon), with stack traces at sample intervals. ### Step 3: Correlate with shell activity ```bash # Most recent zsh history entries (assumes default zsh) fc -l -t '%Y-%m-%d %H:%M:%S' -100 # Or directly: tail -50 ~/.zsh_history ``` The zsh `EXTENDED_HISTORY` format `: <epoch>:<elapsed>;<cmd>` lets you grep for commands run within the incident window. ### Step 4: Synthesize Write a one-paragraph timeline including: - Type of incident (panic / hang / jetsam / power-cycle) - Time T and time-since-boot at T - Top suspects (loaded kexts, busy daemons, low-memory processes) - What recovery looked like (clean reboot / hard power-cycle / GUI returned on its own) ## Common Patterns The reboot-vs-hang one-liner, per-category report counts, the CPU-event offender histogram, and the jetsam victim list are all produced in one pass by the bundled script (`scripts/macos-incident-postmortem.sh` — see "Gather the deterministic signals" above). Read `CLASSIFICATION=`, the `*_COUNT=` keys, `CPU_OFFENDERS:`, and `JETSAM_VICTIMS:` from its output rather than re-running the equivalent shell pipelines. A sudden rise in any category count is a leading indicator before a major hang or panic; offenders/victims that recur are running near their CPU/memory budget. ### Correlate with sleep/wake history ```bash pmset -g log | grep -E 'Sleep|Wake|DarkWake' | tail -50 ``` A spurt of `DarkWake` entries followed by a hang report often means a peripheral or sleep-assertion-holder is the trigger. ## Skip List (Common False Suspects) | Suspect | Why it's usually NOT the cause | |---------|--------------------------------| | `mds`, `mds_stores` (Spotlight) | Heavy I/O is normal after large file changes; rarely panics | | `cloudd` | High network use is normal during iCloud sync | | `bird` (CloudKit) | Same as `cloudd` | | `Time Machine` | Throttles itself; almost never the proximate cause | | `kernel_task` at 100% | This is thermal management *running*, not a bug | If one of these is the only thing visible in your timeline, look harder — the real cause is usually a daemon that *blocked on* one of these, not the daemon itself. ## Agentic Optimizations | Context | Command | |---------|---------| | All deterministic signals + classifier | `bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh" --home-dir "$HOME"` | | Classify against a known T | `MACOS_PM_INCIDENT_EPOCH=<epoch> bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh"` | | Last 5 boots | `last reboot \| head -5` | | Last 5 shutdowns | `last shutdown \| head -5` | | WindowServer log slice | `log show --predicate 'subsystem == "com.apple.WindowServer"' --last 1h --style syslog \| head -200` | ## Quick Reference Key paths, useful `log show` predicates, and time selectors for building the timeline queries: [references/quick-reference.md](references/quick-reference.md). ## Decision Flow ``` Did `last reboot` advance near time T? ├─ YES → Kernel-level event │ ├─ *.panic file present? → Panic; read backtrace │ └─ No panic file → Clean restart (user-initiated or watchdog) └─ NO → Userspace event ├─ *.hang or *.spindump.txt near T? → UI thread hang ├─ *.cpu_resource.diag spike near T? → Daemon CPU storm ├─ JetsamEvent-* near T? → Memory-pressure kill └─ None of the above → Power loss or hard power-cycle ``` ## Error Handling When a command fails or the reports do not reach back to T (permission denied, empty `last reboot`, slow `log show`, rotated reports), open [references/troubleshooting.md](references/troubleshooting.md). ## Related Skills - `launchservices-health` — when the timeline points at `launchservicesd`, dig deeper there - `kitty-session-persistence` — recover terminal state lost during the incident