SkillMedev Alert Tuning
Every alert that fires without requiring human action trains engineers to ignore alerts. Alert fatigue kills real incidents. The goal: every page wakes someone up for a reason they cannot automate away.
The Audit - Run First
Before writing new alert rules, audit what you have.
- Pull the last 30 days of alert history. Calculate: total pages, pages outside business hours, pages that resulted in no action taken.
- Identify flappy alerts: any alert that fires and resolves more than 3 times in a 24-hour window without a corresponding incident.
- Identify cause-based alerts: anything that fires on a system-internal metric (queue depth, pod restart count, CPU) without a direct user-impact correlation.
- Calculate the actionability rate: (pages with a documented remediation / total pages). Below 80% means your alert set is broken.
Symptom vs Cause
Page on symptoms, not causes. Symptoms are what users experience. Causes are what engineers investigate.
- Wrong: alert on CPU above 80%.
- Right: alert on p99 latency above SLO threshold.
- Wrong: alert on pod restart count above 2.
- Right: alert on error rate above 1% for 5 minutes.
For every cause-based alert, ask: 'Would a user notice if this fired and nobody acted?' If no, delete it. If yes, rewrite it as the symptom the user would notice.
SLO-Based Alerting
Tie alerts to error budget burn rate, not raw thresholds.
- Fast burn: burn rate above 14x for 5 minutes (consumes 1 hour of 30-day budget). Page immediately.
- Slow burn: burn rate above 1x for 1 hour. Ticket or warn.
- No burn: no page, regardless of cause-based signals.
This approach eliminates most threshold-tuning debates and makes alert severity self-documenting.
Fixing Flappy Alerts
- Add a 'for' duration to the alert rule. A threshold must be sustained for at least 2-5 minutes before paging.
- Add hysteresis: alert fires at 95% threshold, resolves only when it drops below 85%.
- For bursty metrics, use a rate or average over a longer window (10-15 minutes) instead of instantaneous value.
Routing and Severity
- Route by SLO impact, not by which team owns the metric. The team that can fix it fastest gets the page.
- SEV1/2-level alerts: immediate page.
- SEV3-level alerts: ticket, notify via Slack, no wake-up.
- Never route the same alert to more than one person simultaneously. Shared ownership is no ownership.
Do / Don't
- Do: delete alerts you cannot improve rather than leaving them muted. A muted alert is a lie.
- Do: review alert history quarterly. Thresholds drift as systems scale.
- Don't: add a new alert without deleting or merging one. Alert sets grow monotonically unless you enforce a budget.
- Don't: alert on absence of data without accounting for scrape intervals. Missing data is not always an outage.