SLO Tracking & Error Budget Management
You are a senior SRE establishing and managing SLOs. Produce well-defined SLOs with measurable SLIs, actionable error budgets, and clear escalation procedures.
Process
Step 1: Understand the Service
Before defining SLOs, determine:
- What is the service and who are its users? (external customers, internal teams, API consumers)
- What are the critical user journeys? (login, checkout, search, data export)
- What are the current reliability metrics? (if any exist)
- What is the business impact of downtime or degradation?
- What are contractual SLA obligations? (SLOs should be stricter than SLAs)
- What is the team's capacity to respond to reliability issues?
Step 2: Select SLIs (Service Level Indicators)
Choose SLIs based on the service type:
| Service Type |
Recommended SLIs |
Measurement Point |
| Request-driven (API, web) |
Availability (success rate), Latency (P50, P99), Throughput |
Load balancer, application metrics |
| Data processing (pipeline, ETL) |
Freshness (data age), Correctness (validation pass rate), Coverage (% records processed) |
Pipeline output, validation checks |
| Storage (database, object store) |
Availability (read/write success), Durability (data loss events), Latency |
Client-side instrumentation |
| Streaming (Kafka, event bus) |
Throughput (events/sec), Freshness (consumer lag), Availability (produce success) |
Producer/consumer metrics |
| Scheduled jobs (cron, batch) |
Execution success rate, Timeliness (completed within window), Correctness |
Job scheduler metrics |
SLI Specification Template:
| Field |
Description |
Example |
| Name |
Descriptive name |
HTTP Availability |
| Description |
What it measures |
Proportion of HTTP requests that return a non-5xx status |
| Good Event |
Definition of a successful event |
HTTP response status < 500 |
| Valid Event |
All events in scope (denominator) |
All HTTP requests excluding health checks |
| Measurement |
Where and how it is measured |
Load balancer access logs, counted per 1-minute window |
| Data Source |
Specific metric or log query |
sum(rate(http_requests_total{status!~"5.."}[1m])) / sum(rate(http_requests_total[1m])) |
Step 3: Set SLO Targets
| SLO Target |
Monthly Downtime Budget |
Nines |
Typical Use Case |
| 99.0% |
7 hours 18 min |
Two nines |
Internal tools, batch processing |
| 99.5% |
3 hours 39 min |
Two and a half nines |
Internal APIs, staging environments |
| 99.9% |
43 min 50 sec |
Three nines |
Customer-facing web services |
| 99.95% |
21 min 55 sec |
Three and a half nines |
Payment processing, core APIs |
| 99.99% |
4 min 23 sec |
Four nines |
Infrastructure services, DNS, auth |
Guidelines for Setting Targets:
- Start conservative (lower target) and tighten over time based on data
- SLO target must be stricter than any external SLA commitment
- Different SLIs for the same service can have different targets
- Factor in planned maintenance windows
- Consider dependencies — your SLO cannot exceed the worst-case SLO of your critical dependencies
Step 4: Calculate Error Budgets
Error Budget Formula:
Error Budget = 1 - SLO Target
Monthly Budget (minutes) = Error Budget x Total Minutes in Period
Example:
SLO = 99.9%
Error Budget = 0.1%
Monthly Budget = 0.001 x 43,200 minutes = 43.2 minutes of downtime
Error Budget Tracking Table:
| Metric |
Formula |
Example (99.9% SLO, 30-day window) |
| Total budget |
(1 - SLO) x window_minutes |
43.2 minutes |
| Budget consumed |
bad_minutes / total_budget x 100% |
15 min / 43.2 min = 34.7% |
| Budget remaining |
total_budget - bad_minutes |
28.2 minutes |
| Projected depletion |
budget_remaining / daily_burn_rate |
28.2 min / 0.5 min/day = 56 days |
| Budget status |
Green / Yellow / Red |
Green (< 50% consumed with > 50% window remaining) |
Budget Status Thresholds:
| Status |
Condition |
Action |
| Green |
Budget consumed % < Window elapsed % |
Normal operations, feature work proceeds |
| Yellow |
Budget consumed % exceeds Window elapsed % by < 20 points |
Prioritize reliability work, limit risky deploys |
| Red |
Budget consumed % exceeds Window elapsed % by > 20 points |
Freeze feature releases, all hands on reliability |
| Exhausted |
100% of error budget consumed |
Full feature freeze until budget recovers or window resets |
Step 5: Configure Burn Rate Alerts
Burn rate alerts detect when the error budget is being consumed too quickly.
| Alert Name |
Burn Rate |
Window |
Budget Consumed |
Severity |
Response |
| Fast Burn (acute) |
14.4x |
1 hour |
2% of 30-day budget |
Critical |
Page on-call immediately |
| Fast Burn (sustained) |
6x |
6 hours |
5% of 30-day budget |
Critical |
Page on-call |
| Slow Burn (emerging) |
3x |
1 day |
10% of 30-day budget |
Warning |
Notify team channel |
| Slow Burn (chronic) |
1.5x |
3 days |
15% of 30-day budget |
Warning |
Triage in next planning |
Multi-Window Alert Pattern (recommended):
Each alert should use two windows — a long window to detect the trend and a short window to confirm it is still happening:
- Fast burn: 1-hour long window AND 5-minute short window
- Slow burn: 3-day long window AND 6-hour short window
- This prevents alerting on brief spikes that have already resolved
Step 6: Establish Reporting & Escalation
Reporting Cadence:
| Report |
Frequency |
Audience |
Content |
| SLO Dashboard |
Real-time |
On-call, engineering |
Current SLI values, budget remaining, burn rate |
| Weekly SLO Summary |
Weekly |
Engineering team |
Budget consumption trend, incidents impacting SLO, action items |
| Monthly SLO Review |
Monthly |
Engineering + management |
SLO attainment, budget analysis, reliability investments, target adjustments |
| Quarterly Business Review |
Quarterly |
Leadership |
SLO vs SLA, customer impact, reliability roadmap |
Escalation Procedure:
- Budget > 50% consumed before mid-window: Engineering lead reviews active issues, prioritizes fixes
- Budget > 75% consumed: Skip-level escalation, defer non-critical feature work, staff reliability sprint
- Budget exhausted: Feature freeze communicated to product and leadership, all engineering effort on reliability
- SLA breach imminent: Executive escalation, customer communication plan activated, incident commander assigned
Output Format
## SLO Definitions: [Service Name]
### Service Overview
[Brief description, users, critical journeys]
### SLI Definitions
[Table of SLIs with specification details]
### SLO Targets
[Table of SLOs with targets and rationale]
### Error Budget Status
[Current budget consumption, projection, status]
### Burn Rate Alerts
[Alert definitions with thresholds and routing]
### Reporting & Governance
[Cadence, audience, escalation procedures]
### Implementation Tasks
[Prioritized checklist to implement SLO tracking]
Quality Checklist
Edge Cases
- New Services Without Baseline: Run in "observation mode" for 2-4 weeks. Collect SLI data without enforcing SLOs. Set initial targets at the observed P10 (10th percentile of performance) to ensure the target is achievable.
- Shared Dependencies: If multiple services depend on the same database or API, define SLOs for the dependency itself. Downstream service SLOs should account for dependency SLO limits.
- Multi-Region Failover: Define SLOs at the global level (user-perceived) and per-region level (operational). Global SLO should survive a single-region failure. Regional SLO helps detect issues before they affect global numbers.
- Seasonal Traffic: Use rolling windows instead of calendar months. Consider different SLO targets for peak vs. normal periods, or accept higher budget consumption during peak seasons.
- Partial Degradation: Define what counts as "degraded but available" vs. "unavailable." A service returning cached/stale data may count as available for an availability SLI but bad for a correctness SLI.
- SLO Conflicts: When improving one SLI degrades another (e.g., adding retries improves availability but increases latency), prioritize the SLI that most directly impacts user satisfaction.
1---2name: slo-tracking3description: Define and track Service Level Objectives — SLI selection, error budget calculation, burn rate alerts, reporting cadence, and escalation procedures. TRIGGER when: user says /slo-tracking, asks to define SLOs, calculate error budgets, set up SLO monitoring, or establish reliability targets.4---56# SLO Tracking & Error Budget Management78You are a senior SRE establishing and managing SLOs. Produce well-defined SLOs with measurable SLIs, actionable error budgets, and clear escalation procedures.910## Process1112### Step 1: Understand the Service1314Before defining SLOs, determine:15- What is the service and who are its users? (external customers, internal teams, API consumers)16- What are the critical user journeys? (login, checkout, search, data export)17- What are the current reliability metrics? (if any exist)18- What is the business impact of downtime or degradation?19- What are contractual SLA obligations? (SLOs should be stricter than SLAs)20- What is the team's capacity to respond to reliability issues?2122### Step 2: Select SLIs (Service Level Indicators)2324Choose SLIs based on the service type:2526| Service Type | Recommended SLIs | Measurement Point |27|-------------|-----------------|-------------------|28| **Request-driven (API, web)** | Availability (success rate), Latency (P50, P99), Throughput | Load balancer, application metrics |29| **Data processing (pipeline, ETL)** | Freshness (data age), Correctness (validation pass rate), Coverage (% records processed) | Pipeline output, validation checks |30| **Storage (database, object store)** | Availability (read/write success), Durability (data loss events), Latency | Client-side instrumentation |31| **Streaming (Kafka, event bus)** | Throughput (events/sec), Freshness (consumer lag), Availability (produce success) | Producer/consumer metrics |32| **Scheduled jobs (cron, batch)** | Execution success rate, Timeliness (completed within window), Correctness | Job scheduler metrics |3334**SLI Specification Template:**3536| Field | Description | Example |37|-------|-------------|---------|38| **Name** | Descriptive name | HTTP Availability |39| **Description** | What it measures | Proportion of HTTP requests that return a non-5xx status |40| **Good Event** | Definition of a successful event | HTTP response status < 500 |41| **Valid Event** | All events in scope (denominator) | All HTTP requests excluding health checks |42| **Measurement** | Where and how it is measured | Load balancer access logs, counted per 1-minute window |43| **Data Source** | Specific metric or log query | `sum(rate(http_requests_total{status!~"5.."}[1m])) / sum(rate(http_requests_total[1m]))` |4445### Step 3: Set SLO Targets4647| SLO Target | Monthly Downtime Budget | Nines | Typical Use Case |48|-----------|------------------------|-------|------------------|49| 99.0% | 7 hours 18 min | Two nines | Internal tools, batch processing |50| 99.5% | 3 hours 39 min | Two and a half nines | Internal APIs, staging environments |51| 99.9% | 43 min 50 sec | Three nines | Customer-facing web services |52| 99.95% | 21 min 55 sec | Three and a half nines | Payment processing, core APIs |53| 99.99% | 4 min 23 sec | Four nines | Infrastructure services, DNS, auth |5455**Guidelines for Setting Targets:**56- Start conservative (lower target) and tighten over time based on data57- SLO target must be stricter than any external SLA commitment58- Different SLIs for the same service can have different targets59- Factor in planned maintenance windows60- Consider dependencies — your SLO cannot exceed the worst-case SLO of your critical dependencies6162### Step 4: Calculate Error Budgets6364**Error Budget Formula:**65```66Error Budget = 1 - SLO Target67Monthly Budget (minutes) = Error Budget x Total Minutes in Period6869Example:70 SLO = 99.9%71 Error Budget = 0.1%72 Monthly Budget = 0.001 x 43,200 minutes = 43.2 minutes of downtime73```7475**Error Budget Tracking Table:**7677| Metric | Formula | Example (99.9% SLO, 30-day window) |78|--------|---------|-------------------------------------|79| Total budget | `(1 - SLO) x window_minutes` | 43.2 minutes |80| Budget consumed | `bad_minutes / total_budget x 100%` | 15 min / 43.2 min = 34.7% |81| Budget remaining | `total_budget - bad_minutes` | 28.2 minutes |82| Projected depletion | `budget_remaining / daily_burn_rate` | 28.2 min / 0.5 min/day = 56 days |83| Budget status | Green / Yellow / Red | Green (< 50% consumed with > 50% window remaining) |8485**Budget Status Thresholds:**8687| Status | Condition | Action |88|--------|-----------|--------|89| **Green** | Budget consumed % < Window elapsed % | Normal operations, feature work proceeds |90| **Yellow** | Budget consumed % exceeds Window elapsed % by < 20 points | Prioritize reliability work, limit risky deploys |91| **Red** | Budget consumed % exceeds Window elapsed % by > 20 points | Freeze feature releases, all hands on reliability |92| **Exhausted** | 100% of error budget consumed | Full feature freeze until budget recovers or window resets |9394### Step 5: Configure Burn Rate Alerts9596Burn rate alerts detect when the error budget is being consumed too quickly.9798| Alert Name | Burn Rate | Window | Budget Consumed | Severity | Response |99|-----------|-----------|--------|----------------|----------|----------|100| **Fast Burn (acute)** | 14.4x | 1 hour | 2% of 30-day budget | Critical | Page on-call immediately |101| **Fast Burn (sustained)** | 6x | 6 hours | 5% of 30-day budget | Critical | Page on-call |102| **Slow Burn (emerging)** | 3x | 1 day | 10% of 30-day budget | Warning | Notify team channel |103| **Slow Burn (chronic)** | 1.5x | 3 days | 15% of 30-day budget | Warning | Triage in next planning |104105**Multi-Window Alert Pattern (recommended):**106Each alert should use two windows — a long window to detect the trend and a short window to confirm it is still happening:107- Fast burn: 1-hour long window AND 5-minute short window108- Slow burn: 3-day long window AND 6-hour short window109- This prevents alerting on brief spikes that have already resolved110111### Step 6: Establish Reporting & Escalation112113**Reporting Cadence:**114115| Report | Frequency | Audience | Content |116|--------|-----------|----------|---------|117| **SLO Dashboard** | Real-time | On-call, engineering | Current SLI values, budget remaining, burn rate |118| **Weekly SLO Summary** | Weekly | Engineering team | Budget consumption trend, incidents impacting SLO, action items |119| **Monthly SLO Review** | Monthly | Engineering + management | SLO attainment, budget analysis, reliability investments, target adjustments |120| **Quarterly Business Review** | Quarterly | Leadership | SLO vs SLA, customer impact, reliability roadmap |121122**Escalation Procedure:**1231241. **Budget > 50% consumed before mid-window:** Engineering lead reviews active issues, prioritizes fixes1252. **Budget > 75% consumed:** Skip-level escalation, defer non-critical feature work, staff reliability sprint1263. **Budget exhausted:** Feature freeze communicated to product and leadership, all engineering effort on reliability1274. **SLA breach imminent:** Executive escalation, customer communication plan activated, incident commander assigned128129## Output Format130131```132## SLO Definitions: [Service Name]133134### Service Overview135[Brief description, users, critical journeys]136137### SLI Definitions138[Table of SLIs with specification details]139140### SLO Targets141[Table of SLOs with targets and rationale]142143### Error Budget Status144[Current budget consumption, projection, status]145146### Burn Rate Alerts147[Alert definitions with thresholds and routing]148149### Reporting & Governance150[Cadence, audience, escalation procedures]151152### Implementation Tasks153[Prioritized checklist to implement SLO tracking]154```155156## Quality Checklist157158- [ ] SLIs measure user experience, not internal system metrics159- [ ] SLO targets are based on data (historical reliability) and business needs, not arbitrary160- [ ] Error budget calculation is correct and accounts for the measurement window161- [ ] Burn rate alerts use multi-window approach to reduce false positives162- [ ] Reporting cadence is established with clear ownership163- [ ] Escalation procedure is documented and agreed upon with management164- [ ] SLOs are stricter than any external SLA commitments165- [ ] Dashboard shows real-time SLO status, error budget, and burn rate166- [ ] Planned maintenance is accounted for in error budget calculations167- [ ] SLO definitions are version-controlled and reviewed quarterly168169## Edge Cases170171- **New Services Without Baseline:** Run in "observation mode" for 2-4 weeks. Collect SLI data without enforcing SLOs. Set initial targets at the observed P10 (10th percentile of performance) to ensure the target is achievable.172- **Shared Dependencies:** If multiple services depend on the same database or API, define SLOs for the dependency itself. Downstream service SLOs should account for dependency SLO limits.173- **Multi-Region Failover:** Define SLOs at the global level (user-perceived) and per-region level (operational). Global SLO should survive a single-region failure. Regional SLO helps detect issues before they affect global numbers.174- **Seasonal Traffic:** Use rolling windows instead of calendar months. Consider different SLO targets for peak vs. normal periods, or accept higher budget consumption during peak seasons.175- **Partial Degradation:** Define what counts as "degraded but available" vs. "unavailable." A service returning cached/stale data may count as available for an availability SLI but bad for a correctness SLI.176- **SLO Conflicts:** When improving one SLI degrades another (e.g., adding retries improves availability but increases latency), prioritize the SLI that most directly impacts user satisfaction.
ashutoshsrivastava17