Slo Design

When to Use

• Use when: defining reliability targets for a new service
• Use when: on-call alerts are too noisy or miss real outages
• Use when: engineering team wants to make a data-driven ship/hold decision
• Use when: SLA negotiation requires underlying SLO/SLI data
• Do NOT use for: application-level profiling or performance optimization

SLI / SLO / SLA Hierarchy

SLI  (what you measure)
  └── SLO  (the target you set internally)
        └── SLA  (the commitment you make externally, usually softer)

SLI — a specific metric that measures service behavior:

Good requests / Total requests × 100 = availability SLI
p99 latency measured at load balancer = latency SLI

SLO — a target range for an SLI, measured over a rolling window:

Availability SLO: 99.9% of requests succeed over 30 days
Latency SLO:      p99 < 500ms for 95% of 5-minute windows over 30 days

SLA — an external contract. Typically 0.5–1% below the SLO to allow buffer:

If SLO = 99.9%, then SLA = 99.5% (external commitment)

Never set SLA = SLO. SLAs trigger legal/financial consequences; leave buffer.

SLI Selection by Service Type

Rules:

• 1–3 SLIs per service. More = noise, harder to act on.
• Measure at the user-facing boundary — not deep inside the stack.
• Latency SLI: track p99, not average. Average hides tail pain.

Error Budget

Error budget = 100% − SLO target

Example: SLO = 99.9% → Error budget = 0.1% over 30 days
         = 0.1% × 30 × 24 × 60 min = 43.2 minutes of downtime/month

Error budget policy (run this on every sprint planning)

This policy must be written down and agreed by engineering + product before incidents happen.

Burn-Rate Alerting

Burn rate = how fast the error budget is being consumed relative to expected pace.

Burn rate 1x = consuming budget at exactly the SLO-allowed rate
Burn rate 2x = consuming 2× the allowed rate — budget halved in remaining window

Recommended alert thresholds (Google SRE model)

Prometheus example

# Alert: 14× burn rate over 1 hour (page-level)
- alert: HighErrorBudgetBurnRate
  expr: |
    sum(rate(http_requests_total{job="api", status=~"5.."}[1h]))
    /
    sum(rate(http_requests_total{job="api"}[1h]))
    > (14 * 0.001)   # 14× burn on 99.9% SLO = 0.1% budget
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning at 14× — page on-call"

SLO Dashboard — Minimum Requirements

Every service with an SLO must have a dashboard showing:

□ Current SLI value (30-day rolling window)
□ Error budget remaining (% + absolute time/requests)
□ Error budget burn rate (current 1h and 6h)
□ SLO target line on the SLI graph
□ Recent incidents and their budget impact

Common Mistakes

Anti-Fake-Pass Rules

Before claiming an SLO is "designed", you MUST show:

• SLI defined: metric, measurement point, time window
• SLO target set with explicit reasoning (not arbitrary 99.9%)
• Error budget calculated in absolute units (minutes / requests)
• Burn-rate alerts configured at ≥ 2 thresholds (fast burn + slow burn)
• Error budget policy written and agreed (not just "we'll figure it out")

Reference: gates/anti-fake-pass-gate.md