cloudthinker-ai Alert Fatigue Reduction
Phase 1: Alert Volume Analysis
Gather alert data for the analysis window.
- Total alerts fired: ___
- Total unique alert rules that fired: ___
- Alerts per day (average): ___
- Alerts per on-call shift (average): ___
- Peak alert hour/day: ___
- Off-hours pages (nights/weekends): ___
Phase 2: Alert Classification
Classify every alert rule that fired during the window.
| Alert Rule | Fires (count) | Actionable (%) | Avg Resolution Time | Auto-resolved (%) | Category |
|---|---|---|---|---|---|
Categories:
| Category | Definition |
|---|---|
| Actionable | Required human intervention that prevented or mitigated user impact |
| Informational | Provided useful context but required no action |
| Noisy | Fired frequently without requiring action; contributes to fatigue |
| Redundant | Duplicates another alert for the same condition |
| Stale | Monitors a condition that is no longer relevant |
| Misconfigured | Threshold too sensitive or monitoring wrong metric |
Phase 3: Signal-to-Noise Assessment
- Calculate signal-to-noise ratio:
actionable_alerts / total_alerts * 100 - Target ratio: >70% actionable
- Current ratio: ___%
- Identify top 10 noisiest alert rules
Decision Matrix — Alert Disposition:
| Category | Action |
|---|---|
| Actionable | Keep. Review thresholds for optimization. |
| Informational | Convert to dashboard or log-based notification. Remove paging. |
| Noisy | Adjust thresholds, add dampening, or consolidate with related alerts. |
| Redundant | Delete or merge into parent alert. |
| Stale | Delete after confirming with service owner. |
| Misconfigured | Fix threshold, metric, or query. Revalidate. |
Phase 4: On-Call Impact Assessment
- Survey on-call engineers about alert quality (1-5 scale)
- Measure sleep interruption frequency
- Identify alerts most commonly snoozed or acknowledged-without-action
- Assess on-call handoff notes for recurring complaints
Phase 5: Remediation Plan
For each alert requiring changes:
- Document current configuration
- Define new configuration (threshold, window, dampening)
- Test in non-paging mode for 1 week
- Promote to paging after validation
- Monitor for false negatives
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|---|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |
Output Format
Summary
- Total alerts analyzed: ___
- Signal-to-noise ratio: ___%
- Alerts to remove: ___
- Alerts to tune: ___
- Projected reduction in alert volume: ___%
Action Items
- Delete all stale and redundant alerts
- Tune top 10 noisiest alerts within 2 weeks
- Convert informational alerts to dashboard widgets
- Establish monthly alert hygiene review
- Set team target for signal-to-noise ratio