Incident Notes
Write the note within one working day of the incident, while the release thread is still fresh.
Steps
- Read the release thread from the first message of the deploy to the all-clear.
- Pull the deploy log for the tag:
make deploy-log ENV=prod RELEASE=<tag>. - Write the note in
docs/incidents/<date>-<slug>.mdwith four sections: what customers saw, timeline in UTC, cause, what changes. - State customer impact as numbers from the checkout dashboard: failed checkouts, and minutes above the latency threshold.
- List each follow-up as one line with one owner.
Gotchas
- The timeline uses the timestamps in the deploy log, not the times people posted. Messages lag the events by minutes.
- Name systems, not people. "The promote target does not check the payments queue", not who ran it.