Macos Incident Postmortem

macOS Incident Postmortem

When to Use This Skill

Platform Guard

This skill is macOS-only. /Library/Logs/DiagnosticReports/, kern.boottime, pmset -g log, and the log command are Darwin-specific. Refuse to act if uname -s is not Darwin.

test "$(uname -s)" = "Darwin" || { echo "macos-plugin: not Darwin, refusing"; exit 1; }

Core Expertise

A macOS "incident" can mean any of:

• A kernel panic (system reset by the kernel)
• A WindowServer userspace watchdog timeout (the GUI froze and the user power-cycled)
• A LaunchServices / coreaudiod / mds-stores XPC stall (one daemon dragged the GUI down)
• A jetsam memory-pressure event (the kernel killed apps to reclaim RAM)
• A thermal throttle (CPU clamped to base frequency for minutes)
• A user-initiated force-reboot after a hang (no panic; just lost state)

The first job in a postmortem is distinguishing between actual reboots and GUI hangs. The 2026-04-22 incident that motivated this plugin looked like a crash to the user but last reboot showed no reboot — the kernel was fine, only the GUI stack hung.

The second job is timeline reconstruction: cross-reference Diagnostic Reports, kern.boottime, last reboot, last shutdown, and shell history to answer "what happened around time T?".

Gather the deterministic signals

Run the bundled script to gather every mechanical signal in one pass — reboot-vs-hang detection, per-category DiagnosticReports counts, the CPU-event offender histogram, the jetsam victim list, and the fixed reboot/hang classifier:

bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh" --home-dir "$HOME"

Parse STATUS= and ISSUES: from the output. The script emits CLASSIFICATION= (REBOOT_PANIC / REBOOT_CLEAN / HANG_UI / HANG_CPU / HANG_JETSAM / HANG_OR_POWERCYCLE / UNKNOWN) with a CLASSIFICATION_REASON=, plus PANIC_COUNT= / HANG_COUNT= / JETSAM_COUNT= / CPU_RESOURCE_COUNT= etc., LATEST_PANIC= / LATEST_HANG= paths, and CPU_OFFENDERS: / JETSAM_VICTIMS: histograms.

Pass the incident time T as an epoch to enable the classifier (MACOS_PM_INCIDENT_EPOCH=<epoch> bash …), or supply an alternate report source with --reports-dir <path>. With no incident time the classifier reports UNKNOWN rather than guessing — feed it T once you know it.

The reboot/hang decision tree the script encodes:

Diagnostic report category reference

/Library/Logs/DiagnosticReports/ collects everything macOS thinks is worth keeping; the script classifies each by filename suffix:

Note: Apple migrated most categories to the .ips extension circa Monterey. Older systems and some categories still produce legacy extensions. The script matches by suffix, not by exact filename.

last reboot reads /var/log/wtmp.X rotated logs. On modern macOS, also check the unified log when wtmp has rotated past the incident:

log show --predicate 'eventType == "stateEvent" AND (event == "boot" OR event == "shutdown")' \
  --last 7d --style syslog

Timeline Reconstruction (judgment)

With the deterministic classification and counts in hand, the remaining steps are the agent's job: read the panic backtrace, name suspect kexts, correlate with logs and shell activity, and write the narrative.

Step 1: Inspect the unified log around T

# Adjust the time window to bracket T
log show --start "2026-04-22 08:15:00" --end "2026-04-22 08:25:00" \
  --predicate 'subsystem == "com.apple.WindowServer" OR process == "launchservicesd" OR process == "coreaudiod"' \
  --style syslog \
  | head -500

Common signatures to grep for:

Step 2: Inspect the panic / hang report

The script's LATEST_PANIC= / LATEST_HANG= lines give you the file to read. Key fields in a panic report:

Hang reports are spindump-style: one column per thread of the hung process (typically WindowServer or the offender daemon), with stack traces at sample intervals.

Step 3: Correlate with shell activity

# Most recent zsh history entries (assumes default zsh)
fc -l -t '%Y-%m-%d %H:%M:%S' -100

# Or directly:
tail -50 ~/.zsh_history

The zsh EXTENDED_HISTORY format : <epoch>:<elapsed>;<cmd> lets you grep for commands run within the incident window.

Step 4: Synthesize

Write a one-paragraph timeline including:

• Type of incident (panic / hang / jetsam / power-cycle)
• Time T and time-since-boot at T
• Top suspects (loaded kexts, busy daemons, low-memory processes)
• What recovery looked like (clean reboot / hard power-cycle / GUI returned on its own)

Common Patterns

The reboot-vs-hang one-liner, per-category report counts, the CPU-event offender histogram, and the jetsam victim list are all produced in one pass by the bundled script (scripts/macos-incident-postmortem.sh — see "Gather the deterministic signals" above). Read CLASSIFICATION=, the *_COUNT= keys, CPU_OFFENDERS:, and JETSAM_VICTIMS: from its output rather than re-running the equivalent shell pipelines. A sudden rise in any category count is a leading indicator before a major hang or panic; offenders/victims that recur are running near their CPU/memory budget.

Correlate with sleep/wake history

pmset -g log | grep -E 'Sleep|Wake|DarkWake' | tail -50

A spurt of DarkWake entries followed by a hang report often means a peripheral or sleep-assertion-holder is the trigger.

Skip List (Common False Suspects)

If one of these is the only thing visible in your timeline, look harder — the real cause is usually a daemon that blocked on one of these, not the daemon itself.

Agentic Optimizations

Quick Reference

Key paths

Useful log show predicates

Time selectors

Decision Flow

Did `last reboot` advance near time T?
├─ YES → Kernel-level event
│   ├─ *.panic file present? → Panic; read backtrace
│   └─ No panic file → Clean restart (user-initiated or watchdog)
└─ NO → Userspace event
    ├─ *.hang or *.spindump.txt near T? → UI thread hang
    ├─ *.cpu_resource.diag spike near T? → Daemon CPU storm
    ├─ JetsamEvent-* near T? → Memory-pressure kill
    └─ None of the above → Power loss or hard power-cycle

Error Handling

Related Skills

• launchservices-health — when the timeline points at launchservicesd, dig deeper there
• kitty-session-persistence — recover terminal state lost during the incident