Incident Response

Incident response

Establish the situation

1. Record the current time, impact, affected scope, and incident start time.
2. Identify what is confirmed, what is inferred, and what remains unknown.
3. Define the expected behavior and the observed deviation.
4. Determine whether impact is growing, stable, or recovering.

Build the timeline

Collect timestamps for:

• First customer or monitoring signal.
• Deployments and configuration changes.
• Dependency failures and recovery events.
• Operator actions and automated mitigations.
• Changes in traffic, capacity, latency, errors, and resource pressure.

Normalize timestamps before drawing conclusions.

Manage hypotheses

For every hypothesis record:

• Evidence that supports it.
• Evidence that contradicts it.
• The smallest safe test that could disprove it.
• Confidence and remaining unknowns.

Prefer discriminating tests over collecting more undirected data.

Mitigate safely

• Prefer reversible, well-understood actions.
• State blast radius and rollback before acting.
• Do not combine several mitigations when one can be evaluated independently.
• Preserve diagnostic evidence when possible.
• Separate immediate impact reduction from permanent remediation.

Close the investigation

Produce:

1. Impact summary.
2. Confirmed timeline.
3. Root cause with supporting evidence.
4. Trigger and contributing factors.
5. Mitigation and recovery.
6. Detection and response gaps.
7. Prioritized follow-up work with owners when known.