Alerting Doctrine

alerting-doctrine

When to use

Use when classifying an alert, reviewing an alert set, or deciding whether a condition should wake a human. Provider-neutral: the classes below hold whichever tool evaluates the rule.

Do NOT use when:

• Establishing which signals exist at all (use logging-monitoring first — you cannot alert on a signal the project does not have)
• Adjudicating a go/no-go verdict (use operational-readiness)
• Running an incident once it has started (use incident-commander)
• Building the dashboard the alert links to (use dashboard-design, grafana)

The three classes

Every alert is exactly one of these. The class is decided by what the recipient must do, never by how alarming the condition sounds.

What earns a page — all three, as a condition rather than a preference:

1. User-visible or irreversible. Users are affected now, or damage accumulates while nobody looks (data loss, budget burn, a queue filling toward a ceiling).
2. Actionable now. A named human can do something about it at 3am. "The upstream provider is down" is not actionable unless failover is manual.
3. Not self-clearing. A condition that resolves itself before a human can log in is an info with a trend, not a page.

Fail any one → it is action or info.

What a misclassification costs. Too eager: alert fatigue, and the page that mattered arrives in a stream of pages that did not. Fatigue is not an inconvenience — it is the mechanism by which a real outage is missed, and it is caused by pages that were never actionable. Too shy: the condition sits in a channel nobody reads until a user reports it. The asymmetry is real but bounded in both directions, which is why the three conditions above are stated as a test and not as taste.

A page is malformed without all three fields

A PAGE THAT CANNOT NAME AN OWNER, A RUNBOOK AND A FIRST DIAGNOSTIC STEP
IS A CONFIGURATION DEFECT, NOT A JUDGEMENT CALL.

This is checkable from the alert definition alone, which is what makes it a defect rather than an opinion. action and info alerts do not carry the obligation — nobody is being woken.

The runbook contract — deliberately lean

A runbook worth paging into carries five things. More than this and it rots faster than the system changes; fewer and it is a link that wastes the minutes it was supposed to save.

1. What this alert means — one sentence, in terms of user impact.
2. First diagnostic step — the one command, query or dashboard to open. The same value the alert definition carries, so the two cannot drift.
3. The two or three most common causes, each with how to confirm it.
4. Mitigation vs fix — how to stop the bleeding, marked separately from how to repair the cause. Under a page, mitigation is the goal.
5. Escalation — who is next, and the condition that triggers going to them.

Explicitly not in a runbook: architecture background, a full topology diagram, anything the reader could get later. A runbook is read under time pressure by someone who was asleep.

→ Once an incident is open, incident-commander owns the roles, the update cadence and the post-mortem obligation. This skill stops at the moment the page fires.

Procedure: Classify and validate an alert

1. Confirm the signal exists. An alert on a signal logging-monitoring scored unavailable or unknown cannot fire meaningfully. Fix the signal first.
2. Apply the three page conditions. User-visible or irreversible, actionable now, not self-clearing. All three → page. Otherwise → action if a human must eventually act, else info.
3. For a page, check the three mandatory fields. Any missing → verdict malformed-alert: missing-<field>. Do not downgrade the class to dodge the obligation; a page whose fields cannot be named is either a defect to fix or was never a page.
4. For a page, check the runbook against the five-item contract, and confirm its first diagnostic step matches the alert's.
5. Record the verdict per alert — valid-page-alert, valid-action-alert, valid-info-alert, or malformed-alert: <reason>.

Validate

• Verify every page names an owner resolving to a single human.
• Verify every page links a runbook that exists.
• Verify every page carries a first diagnostic step identical to its runbook's.
• Confirm no alert was reclassified downward purely to avoid the three fields.
• Confirm every alert's underlying signal is present, not unknown.
• Confirm the alert set contains no page that is self-clearing.

Output format

1. One row per alert — name, class, verdict.
2. For each malformed-alert, the missing field named explicitly.
3. The pages, with owner, runbook pointer and first diagnostic step.

Examples

Contract fixtures. Three pairs, one per mandatory field; each pair differs only in the field under test, and each case states the verdict it must produce. Contract evidence for review, not an executable suite.

Missing owner — malformed vs clean

# malformed
alert: checkout_error_rate_high
class: page
runbook: docs/runbooks/checkout-errors.md
first_diagnostic_step: "open the checkout error-rate panel, last 30m"
verdict: "malformed-alert: missing-owner"

# clean — only the owner is added
alert: checkout_error_rate_high
class: page
owner: payments-on-call
runbook: docs/runbooks/checkout-errors.md
first_diagnostic_step: "open the checkout error-rate panel, last 30m"
verdict: valid-page-alert

Missing runbook — malformed vs clean

# malformed
alert: db_pool_saturation
class: page
owner: platform-on-call
first_diagnostic_step: "check db_pool_in_use / db_pool_size"
verdict: "malformed-alert: missing-runbook"

# clean — only the runbook is added
alert: db_pool_saturation
class: page
owner: platform-on-call
runbook: docs/runbooks/db-pool.md
first_diagnostic_step: "check db_pool_in_use / db_pool_size"
verdict: valid-page-alert

Missing first diagnostic step — malformed vs clean

# malformed
alert: queue_wait_time_high
class: page
owner: platform-on-call
runbook: docs/runbooks/queue-wait.md
verdict: "malformed-alert: missing-diagnostic-step"

# clean — only the first diagnostic step is added
alert: queue_wait_time_high
class: page
owner: platform-on-call
runbook: docs/runbooks/queue-wait.md
first_diagnostic_step: "open the queue wait-time panel; compare to enqueue rate"
verdict: valid-page-alert

Gotcha

• Severity labels are not classes. critical in a tool's own vocabulary says nothing about whether a human must act now.
• A page nobody can action is the fastest route to ignoring all pages.
• Reclassifying a page to action because the owner is unknown hides the defect instead of fixing it.
• An alert on a threshold with proposed provenance is alerting on a guess — see logging-monitoring.
• Queue depth alerts fire late; queue wait time fires in time.

Do NOT

• Do NOT ship a page without an owner, a runbook and a first diagnostic step.
• Do NOT page on a self-clearing condition.
• Do NOT page on a signal scored unknown.
• Do NOT put architecture background in a runbook — it is read under pressure.
• Do NOT let the alert's first diagnostic step and the runbook's disagree.

Auto-trigger keywords

• alert
• alerting
• page
• on-call
• runbook
• alert fatigue
• what earns a page