Incident Hotfix

Incident Hotfix

When to use

• Production is broken, degraded, or actively paging.
• A revert or tiny patch will restore service faster than a redesign.
• The user says "hotfix", "SEV", "outage", or "rollback".

When not to use

• Feature work on a healthy system with normal review timelines.
• Post-incident improvements once mitigation is done (schedule separately).
• Speculative hardening without an active or imminent user impact.

Assumptions

• Access to deploy/rollback mechanisms, logs, metrics, and the hotfix branch process the team uses.
• Communication channel for status may exist; follow the team's incident lead if present.
• Do not run destructive data repairs (DROP, mass DELETE, force-push to main, prod secret rotation storms) without explicit confirmation from the incident owner.
• Do not disable auth, open security groups widely, or bypass CI without approval.

Workflow

1. Stabilize: confirm impact, recent changes, and whether revert is safer than forward fix.
2. Choose the smallest mitigation (feature flag off, revert, config fix, one-line patch).
3. Bound blast radius: target canaries/single region when possible; avoid global risky changes.
4. Ship with a clear rollback path documented before or with the deploy.
5. Verify recovery on golden signals; only then file follow-up work for root cause/refactors.

Steps

1. Triage — Symptom, scope (who/what/%), onset time, last deploy/change. Prefer timeline from metrics/logs over guesses.
2. Mitigation choice — Revert > config/flag > tiny forward fix. Avoid refactors in the hotfix PR.
3. Blast radius — Limit audience (canary, single service). No drive-by dependency upgrades.
4. Rollback plan — Exact command or previous artifact version; who can execute it.
5. Verify — Error rate, latency, and the failing user journey return to baseline.
6. Handoff — Write a short timeline + follow-ups (proper fix, tests, postmortem). Do not expand the hotfix.

Success criteria

• User-facing impact is mitigated or clearly reduced.
• Change set is minimal and free of unrelated refactors.
• Rollback path is documented and feasible.
• Verification used real signals (metrics/logs/journey), not only "deploy succeeded".
• Destructive or security-weakening actions were confirmed or avoided.

Out of scope

• Full postmortem facilitation and action-item programs.
• Long-term architecture rewrites during the incident window.
• Customer comms strategy beyond brief factual status if you are not the comms owner.