gesh75 Dual-signal observability
Do not use for capacity forecasting or flow forensics. Not for log RCA (use a sanitize-first analyzer).
Procedure
- Pick two independent planes: interface discards AND synthetic probe; BGP session AND prefix count; syslog AND streaming telemetry.
- Page only on AND. Slack on OR. Never page on a single SNMP counter.
- Dedup window 15 minutes. Flaps inside the window are one incident.
- Every alert names the next command to run and the rollback if a recent change exists.
- Verify by injecting a fault in the lab and confirming both signals fire.
alert: site-2401-wan-down
all_of:
- bfd.color.mpls == down
- probe.voice.mos < 3.5
for: 2m
Anti-patterns
- Never page on CPU 80% alone.
- Do not average away a 5-second microburst that drops voice.
Safety
Alert rule edits are reviewed. Collectors stay read-only against devices.