Production Incident

Production Incident

The on-fire workflow. Different from debug-deep (dev-time investigation) — this one assumes users are affected RIGHT NOW and prioritizes blast-radius reduction before root cause.

Auto-invocation triggers

• User says "prod is down", "users are reporting X", "Sentry is firing"
• ship-it Phase 4 detected new Sentry issues post-deploy
• Manual escalation from a regular bug that turned out to be production-impacting
• Sentry alert webhook (if wired)

Workflow — sequential, blast-radius first

Phase 1 — Triage (ALWAYS, FIRST)

Invoke sentry to pull the incoming event picture:

• Issue count + frequency in last 15 minutes
• Affected users / orgs / regions
• First seen timestamp (correlate with recent deploys)
• Stack trace top frame

Decide blast-radius:

• Single user / single feature → Phase 2 (investigate)
• Multiple users / one feature → Phase 2 + parallel Phase 3 (consider rollback)
• Multiple users / multiple features → SKIP to Phase 3 (rollback first, debug after)

Phase 2 — Root cause (if blast-radius limited)

Invoke debug-deep:

• Hypothesis tree from the Sentry stack trace
• Tracer agent for evidence walk
• Bisect to introducing commit (almost certainly the most-recent deploy)

Phase 3 — Stop the bleeding (if blast-radius high OR Phase 2 confirms recent deploy)

Two options based on severity:

Option A — Rollback (preferred for high blast-radius):

• Invoke ship-it in rollback mode: revert to previous tag, deploy
• Communicate via incident-response (Linear ticket + Slack post)
• Continue to Phase 4 to fix forward

Option B — Hotfix (for limited blast-radius with known fix):

• Implement the fix
• Skip normal merge gates: invoke ship-it with --hotfix (still NOT --admin)
• Verify in Phase 5

Phase 4 — Coordinate (always)

Invoke incident-response to:

• Create Linear incident ticket
• Cross-reference Sentry issue ID
• Pin a Slack/Discord post if applicable
• Track timeline (alert time → triage → action → resolve)

Phase 5 — Verify resolution (always)

• Wait 5-10 min after action
• Invoke sentry again — confirm new event rate dropped to zero or pre-incident baseline
• Hit affected endpoint manually to confirm
• Update Linear ticket with resolution

Phase 6 — Post-mortem (always, mandatory)

Invoke adr-write with post-mortem template:

• Timeline (alert → resolved)
• Root cause
• Why monitoring didn't catch it earlier
• Why it slipped through tests / CI
• Action items: tests to add, monitors to add, gates to tighten
• Revisit when: when the action items are complete

Then invoke knowledge-loop to capture for future similar incidents.

Reconciliation

PRODUCTION INCIDENT — <issue summary>

Detected:    <timestamp> via <source>
Blast:       <users/scope affected>
Root cause:  <one-liner from Phase 2>
Action:      <rollback to v<X> | hotfix v<Y>>
Resolution:  <timestamp>, took <minutes>
Sentry:      <event rate before / after>

Linear:      <ticket URL>
Post-mortem: ADR-NNNN

Action items: <count> (tests:N, monitors:N, gates:N)

Outputs / Evidence

• Triage report
• Root cause + evidence
• Rollback or hotfix deploy proof
• Linear ticket
• Post-mortem ADR with action items
• Sentry resolution confirmation

Failure / Stop Conditions

• Sentry inaccessible → cannot triage; surface immediately, fall back to manual user-report investigation
• Rollback fails (deploy broken too) → ESCALATE — page humans; this skill cannot fix infrastructure failures
• Phase 6 ADR cannot be written without root cause → block; do not declare resolved without understanding why

Memory Hooks

• Read prior incidents on the same component to detect recurring patterns
• Write incident outcome with full timeline; trend across months reveals fragile areas worth structural refactors