Alert Hysteresis

alert-hysteresis

Analyzes how long alerts have actually stayed firing in the past (via Prometheus's ALERTS{alertname=...} series) and recommends a for: duration that would filter out short-lived, non-actionable firings while still catching the ones that matter. This is a judgment-assisted, interactive tool, not a static analyzer — every run depends on real, non-reproducible production history.

When to use this skill

• On-call has flagged an alert as noisy/flappy and you want a data-driven for: recommendation instead of guessing.
• You're auditing a rules file's existing for: values against how the alert has actually behaved in production.
• Do not wire this into CI as a blocking check: its output depends on live Prometheus data that CI can't reproduce deterministically, and the right response to a recommendation is a human decision, not an auto-applied diff (except deliberately, via --fix, described below).

Prerequisites

• A reachable Prometheus (or Thanos/Cortex/Mimir with a Prometheus- compatible /api/v1/query_range endpoint) that has been actually evaluating the alerting rules in question, so the ALERTS metric has history to query.
• Retention matters: this tool can only see as far back as --timeframe and as far back as Prometheus has actually retained samples. A --timeframe=30d against a Prometheus with 15d retention will silently undercount — always sanity-check retention before trusting a "no data" or thin-sample result.

Setup

go build -o bin/alert-hysteresis ./cmd/alert-hysteresis
# or: go install github.com/conallob/o11y-analysis-tools/cmd/alert-hysteresis@latest

Usage

alert-hysteresis [options]

Typical invocations

# Survey all alerts over the last 7 days (default)
alert-hysteresis --prometheus-url=http://prometheus:9090

# Focus on one noisy alert over the last day
alert-hysteresis --prometheus-url=http://prometheus:9090 --alert=HighErrorRate --timeframe=24h

# Compare against configured values, only flag >30% mismatches
alert-hysteresis --prometheus-url=http://prometheus:9090 --rules=./alerts.yml --threshold=0.3

# Apply recommendations directly (review the diff before committing!)
alert-hysteresis --prometheus-url=http://prometheus:9090 --fix --rules=./alerts.yml --target-percentile=0.5

Reading the output

Per alert: firing count, average/median/P75/P90/min/max durations, the configured for: (if --rules given), and either a RECOMMENDATION with reasoning and prevented-alert count, or a confirmation that the current value is acceptable. A closing summary counts how many alerts need adjustment. Exit code 1 if any alert needs adjustment (or, without --rules, whenever any alert simply has a nonzero recommendation to report against — read the summary line to disambiguate "found N alerts with recommendations" from "found N alerts that need hysteresis adjustment").

Agent workflow

1. Always pass --rules when one exists — without it you only get raw statistics, not an actionable "current vs. recommended" comparison.
2. Start with the default --target-percentile=0.3/--threshold=0.2 and discuss the tradeoff with the user before changing them: a higher percentile means a longer for: (fewer alerts, slower detection); a lower one means faster detection but less spurious-alert filtering.
3. Prefer printing recommendations and letting a human review the diff over using --fix directly, unless the user has explicitly asked for the change to be applied — this tool is tuning production alerting behavior, and a bad for: value can hide a real incident.
4. If results look empty or thin, check Prometheus retention against --timeframe before concluding the alert genuinely never fires long enough — that's a job better suited to stale-alerts-analyzer anyway.