Identify the PoMos
- If the pomos are local in a folder, use those.
- If they're external, create a new folder called
YYYYMMDD_POSTMORTEMS/ and copy their MD version there. This might lose information, but ensure the relevant info is transported here, locally. You could run the "postmortem generator" skill on a locally curled page.
- If there is ONE, refuse to run this.
- If there is 2-3, suggest the user it's worth using a plurality of these.
Action
Check all the Post Mortems in the folder and identify these informations:
- Timestamp of outage start.
- Timestamp of outage identified by human and being worked on (if available).
- Timestamp of first Mitigation going into place (if available)
- Timestamp of outage end.
- Product used (eg Cloud, Youtube). Products could have a taxonomy (eg
Cloud::GCE::PersistentDisk, Youtube::Playlists)
- RootCause class (HUMAN_ERROR, CONFIG_CHANGE, EXTERNAL_CAUSE, ..).
- Outage size, how bad it was (TEST, NEGLIGIBLE, SMALL, MEDIUM, BIG, CATASTROPHIC).
Build a pomo_stats.csv with this data, one line per incident
filename,title,outage_start,outage_detected,outage_mitigated,outage_ended,root_cause,size
folder/20260101-millennium-bug.md,Millennium Bug hit us!,20260101-09:00:00,,,20260101-12:42:00,HUMAN_ERROR,NEGLIGIBLE
Crunching the data
Once you have the CSV with raw data, you can write some simple code which does the following:
- Decide what YOUR_DESIGNATED_FOLDER is.
- Create or Update
POMO_AGGREGATED.md under YOUR_DESIGNATED_FOLDER/.
- Caculate for every incident:
outage_duration (end - start)
time_to_detect (detect - start)
time_to_mitigate (mitigate - detect).
detection_rate (ratio between time_to_detect / outage_duration ). This is indicative of how fast we're to OBSERVE an outage vs FIX it and tells us a lot about our organization.
- Do an aggregation of all stats except detection_rate (its a percentage on varying numbers, so it's comparing watermellons to cherries).
- Do a subjective analysis of the detection_rate.
- Create breakdown by YEAR and PRODUCT.
- Consider using image generation scripts to create simple graphs.
Lessons learnt
- See recurring patterns and add a H2 paragraph on
## recurring patterns: Is there anything we can learn from this aggregation? Is this a mistake we keep on doing? Is there a bug which looks like we're not prioritizing? Is there a number of bugs who all look like the same? This is the step where we can actually identify schemas in this plurality.
- Is there some other lesson we learn from this plurality? Write about it in the
## Conclusions.
1---2name: postmortem-aggregator3description: 🐉 [SRE] To be used when you have a folder containing N Post Mortem files. This will help crunch data and maintain/update a POMO_AGGREGATED.md file4---56# Identify the PoMos78* If the pomos are local in a folder, use those.9* If they're external, create a new folder called `YYYYMMDD_POSTMORTEMS/` and copy their MD version there. This might lose information, but ensure the relevant info is transported here, locally. You could run the "postmortem generator" skill on a locally curled page.10* If there is ONE, refuse to run this.11* If there is 2-3, suggest the user it's worth using a plurality of these.1213## Action1415Check all the Post Mortems in the folder and identify these informations:16171. Timestamp of outage start.181. Timestamp of outage identified by human and being worked on (if available).191. Timestamp of first Mitigation going into place (if available)201. Timestamp of outage end.211. Product used (eg Cloud, Youtube). Products could have a taxonomy (eg `Cloud::GCE::PersistentDisk`, `Youtube::Playlists`)221. RootCause class (HUMAN_ERROR, CONFIG_CHANGE, EXTERNAL_CAUSE, ..).231. Outage size, how bad it was (TEST, NEGLIGIBLE, SMALL, MEDIUM, BIG, CATASTROPHIC).2425Build a `pomo_stats.csv` with this data, one line per incident2627```28filename,title,outage_start,outage_detected,outage_mitigated,outage_ended,root_cause,size29folder/20260101-millennium-bug.md,Millennium Bug hit us!,20260101-09:00:00,,,20260101-12:42:00,HUMAN_ERROR,NEGLIGIBLE30```3132## Crunching the data3334Once you have the CSV with raw data, you can write some simple code which does the following:35361. Decide what YOUR_DESIGNATED_FOLDER is.371. Create or Update `POMO_AGGREGATED.md` under YOUR_DESIGNATED_FOLDER/.381. Caculate for every incident:39 * `outage_duration` (end - start)40 * `time_to_detect` (detect - start)41 * `time_to_mitigate` (mitigate - detect).42 * `detection_rate` (ratio between time_to_detect / outage_duration ). This is indicative of how fast we're to OBSERVE an outage vs FIX it and tells us a lot about our organization.431. Do an aggregation of all stats except detection_rate (its a percentage on varying numbers, so it's comparing watermellons to cherries).441. Do a subjective analysis of the detection_rate.451. Create breakdown by YEAR and PRODUCT.461. Consider using image generation scripts to create simple graphs.4748## Lessons learnt4950* See recurring patterns and add a H2 paragraph on `## recurring patterns`: Is there anything we can learn from this aggregation? Is this a mistake we keep on doing? Is there a bug which looks like we're not prioritizing? Is there a number of bugs who all look like the same? This is the step where we can actually identify schemas in this plurality.51* Is there some other lesson we learn from this plurality? Write about it in the `## Conclusions`.