Slo And Alerting

SLO and Alerting

Purpose

Create a measurable user expectation and an operating policy that changes decisions. Then alert early enough, and only when a person has a time-sensitive action.

An SLI is any carefully defined quantitative measure of service level—not necessarily a ratio. An SLO is a target or range on an SLI under declared conditions. An SLA is an agreement carrying explicit consequences. Error-budget ratios are powerful for event-based reliability objectives, but freshness, durability, correctness, throughput and distribution objectives may need other semantics.

Workflow

1. Start from user journeys and consequences

Identify users (including services), critical journeys, harm from failure/latency/staleness or incorrectness, and dependency assumptions. Choose the few objectives that can resolve a real priority conflict. Do not turn every measurable resource into an SLO.

2. Write the SLI specification

Record:

• population and valid-event/time denominator;
• good/bad classification or numerical measurement;
• start/end events and observation point;
• aggregation/window and calendar versus rolling semantics;
• exclusions, unknown/missing data and low-traffic behavior;
• labels/cohorts and weighting;
• source, query, retention and backfill/change policy.

Measure near the user's boundary where practical, while using internal business signals for correctness invisible at the edge. Client telemetry may be sampled or unavailable; document the proxy gap rather than claiming a boundary is always authoritative.

3. Negotiate target and policy

Base targets on user needs, dependency promises, business risk, attainable architecture, cost and observed distribution—without simply adopting current performance. A tighter internal objective than an external SLA often supplies detection/remediation and semantic margin, but “strictly tighter” is not a mathematical requirement: definitions/windows may differ and other controls may supply margin. Reconcile them explicitly.

Avoid 100% availability objectives for failure-prone serving paths unless the scope and consequence genuinely require it. Some correctness/durability invariants can legitimately target zero tolerated events; manage them as safety/data-integrity controls rather than pretending all objectives need a spendable budget.

Define what budget states change: release risk, reliability work, escalation, exemption authority and recovery. A budget is not permission to cause outages deliberately.

4. Derive alerts from response urgency

Page when:

• impact or a predictive hazard is urgent enough to require action before business hours;
• a person has a safe action or escalation now;
• automation cannot handle it fully;
• ownership, runbook and expected response are explicit.

User symptoms are strong paging signals, but “never page on causes” is unsafe. Impending irreversible data loss, certificate expiry inside response lead time, disk exhaustion or a stateful quorum loss can justify predictive paging before users fail. Conversely, a user-visible low-severity condition may only need a ticket.

Use multi-window burn-rate alerts for high-volume ratio SLOs. Use direct deadline/hazard, synthetic probes, minimum-event logic or manual aggregation where burn ratios are ill-conditioned.

5. Validate the monitoring system

Test recording and alert rules with controlled traffic/faults; verify labels, absent data, counter resets, delayed ingestion, partial monitoring outage and alert routing. A page is a production interface and needs version control, review and tests.

Inspect the deployed Prometheus/rule evaluator version, scrape/evaluation intervals, exporter and Java instrumentation library/configuration before recommending syntax or changing buckets. The examples are Prometheus rule fragments, not Java code or a ready-to-load rule file; no Java upgrade or dependency addition is implied. Return the SLI population/query, target/window, budget interpretation, actionable alert policy and checks performed. Keep a safety verdict conditional when traffic, coverage or rule-evaluation evidence is missing.

6. Operate and retire

Review firings by precision, recall, time-to-detect, time-to-action and user impact. Merge correlated pages, automate repeatable remediation, and downgrade or remove alerts that cannot drive action. “Never fired” can mean rare critical coverage, not automatic deletion; exercise it and verify assumptions.

Error-budget arithmetic

For an event-based objective target (S):

[ e_b=1-S,\qquad burn=\frac{e_{observed}}{e_b} ]

For (0<S<1), let (V_w) be valid events in alert window (w), contained in reporting period (T), and (V_T) the valid events in that period. The fraction of that period's total allowed bad events consumed in (w) is:

[ f=b\frac{V_w}{V_T} ]

Only when (V_w/V_T\approx w/T) does this reduce to (f\approx bw/T). With stable event rate and a fixed full-period budget, (T/b) projects time to spend that full budget; it is not time to exhaust today's remaining budget or a guarantee for a rolling window. An unfinished calendar period needs a stated traffic forecast for (V_T); rolling budgets also change as prior events leave. Preserve population/classification across windows and label forecasts as estimates. Time-based SLIs use their own valid-time denominator; request failures are not “minutes unavailable” under variable traffic.

Classification decisions

There is no universal “4xx excluded” or “shed is neither good nor bad.” Encode the user contract. Keep overload classifications distinct so protection is visible even when it counts against availability.

Alert routing framework

Failure modes

Anti-patterns

SLO equals current dashboard: rewards existing implementation and hides user need.

All SLIs are good/valid ratios: freshness gauges, distributions and durability can need different objectives; ratios are especially useful for budget-based availability/latency.

SLA margin by percentage only: differing populations, windows, exclusions and measurement points can consume or create more risk than the numeric gap.

Cause-versus-symptom dogma: route by urgency, actionability and irreversible risk.

No-data equals good: explicitly distinguish absent telemetry, zero denominator and legitimate inactivity.

Budget exhaustion automatically freezes everything: apply a pre-agreed risk policy with exceptions, ownership and business authority; security/safety fixes must not be blocked.

Cross-skill routing

• SLI and error budgets — read when defining populations, targets, budget policy or migrating a measurement.
• Alerting design — read when choosing paging urgency, low-volume handling, missing-data policy or alert ownership.
• Burn-rate rules and templates — read when deriving thresholds or writing/reviewing Prometheus rules.
• metrics-and-cardinality for metric schemas and costs.
• latency-statistics for distributions/quantiles.
• structured-logging and distributed-tracing-design for diagnosis.
• capacity-planning and rate-limiting-and-load-shedding for remediation.

Authoritative references

• Google SRE: Service Level Objectives
• Google SRE Workbook: Implementing SLOs
• Google SRE Workbook: Alerting on SLOs
• Prometheus alerting rules