harishkotra Production Incident Debugger
Debug and fix a live production incident fast, methodically, and safely.
When to use
- A production service is failing (errors, latency, crashes) and needs diagnosis now.
- You have access to logs, metrics, traces, or the codebase.
Procedure
- Establish impact: what's broken, how many users, since when. This drives urgency.
- Get the signal: pull the error logs, stack traces, and the deploy/change that preceded the break.
- Reproduce if possible; if not, work from the evidence you have.
- Form one hypothesis, test it, and confirm before acting.
- Ship the smallest safe fix or rollback, then verify in production.
- Write the postmortem: root cause, fix, and how to prevent recurrence.
Pitfalls
- Don't change multiple things at once or you can't know what fixed it.
- Don't skip the rollback option; a revert is often the fastest safe fix.
Verification
- The incident is resolved (error rate/impact back to baseline) and the root cause is documented.