DevOps SRE Operations
Agent Workflow (MANDATORY)
Before applying this method to a real system, spawn in parallel, one spawn_agent call each:
- explore-codebase - find existing SLO documents, alert rules, runbooks, postmortems, on-call and change records in the repo
- research-expert - verify any figure or tool option against the primary source (sre.google, dora.dev, prometheus.io) before relying on it
- mcp__context7__query-docs - only where a library or tool is involved (Prometheus rule syntax, Alertmanager); the method itself has no library
Then read the live state read-only (current alerts, recent changes, budget remaining) before proposing anything.
After writing files in a repo, run sniper.
Sources and currency (checked 2026-11-10)
Method statements trace to the Google SRE Book and SRE Workbook (online editions), DORA (dora.dev), PagerDuty's public incident response documentation, Prometheus documentation and principlesofchaos.org. Each reference ends with the URLs read and a "Not verified" list: figures that are Google-scale (on-call staffing, response times) are labeled as such; defaults proposed by this skill (update cadence, patch cadence, toil log length) are labeled as the skill's own.
Overview
| Area |
Decision in one line |
| SLO |
100% is the wrong target; SLI = good events / total events; error budget = 1 - SLO; an SLO without a signed error budget policy has no teeth |
| Alerting |
Page on budget burn: 14.4x over 1 h and 5 min, 6x over 6 h and 30 min; ticket on 1x over 3 d and 6 h (99.9% SLO, 30 days); short window = 1/12 of the long one |
| Incidents |
Declare early; separate IC, ops, communications; mitigate before root cause; hand off command explicitly |
| On-call |
Pages are urgent and actionable; at most about 2 incidents per shift; sustainable rotation; handoff in writing |
| Postmortem |
Blameless, quantified impact, timeline in UTC, contributing factors, actions with type and one owner |
| Change |
Small, staged, canary first; rollback criteria written before the change; budget exhausted = freeze except P0 and security |
| Production actions |
Always ACTION / EFFECT / PROOF / WAIT-FOR-GO; the agent proposes, a human says go |
| Toil |
Keep below 50% of time; measure, then design away, automate or document |
| DORA |
Five metrics (lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate); MTTR is the old name |
Critical Rules
- Measure before promising - an SLO needs a defined SLI, a measurement point, an owner and a policy (slo-sli-error-budget.md)
- Alert on user-visible budget burn, with both windows - never a bare
for: on a short window (burn-rate-alerting.md)
- Mitigate first, then find the cause - stop the bleeding, preserve evidence (incident-response.md, troubleshooting-method.md)
- One change at a time, one commander - only the operations lead changes production during an incident
- Rollback criteria are decided before the change, in numbers, in the change request (change-rollout.md)
- Every production action uses ACTION / EFFECT / PROOF / WAIT-FOR-GO and waits for the go; the agent never rolls back, fails over, restarts, shifts traffic or lifts a freeze on its own initiative
- Blameless, with owners - postmortems fix systems, each action item has a type, an owner and a ticket (postmortem.md)
- No invented figures - a number not in a verified source is labeled as this skill's default or marked not verified; derive burn-rate and budget values with the formulas, show the arithmetic
- Never print secrets - runbooks and incident documents name the store path, not the value (devops-secrets)
Architecture
reliability/
├── slos/<service>.md # filled slo-definition.md, one per service
├── policy/error-budget-policy.md # signed by the product owner
├── alerts/<service>.rules.yml # recording + burn-rate alert rules, promtool-checked
├── runbooks/<alert-or-operation>.md
├── incidents/<id>.md # live document, then the postmortem
├── changes/<chg-id>.md # change request and rollout plan
├── readiness/<service>-prr.md
└── oncall/handoff-<date>.md
Versions
Pinned versions:
google-sre-book: "Site Reliability Engineering, O'Reilly 2016, web edition at sre.google/sre-book (page copyright 2017), read 2026-11-10"
google-sre-workbook: "The Site Reliability Workbook, O'Reilly 2018, web edition at sre.google/workbook, read 2026-11-10"
dora: "five-metric software delivery model; dora.dev guide updated 2026-01-05, history page updated 2026-01-02"
prometheus-rules: "rule file syntax from prometheus.io docs (latest), read 2026-11-10; promtool not run (not installed on authoring host)"
pagerduty-response-docs: "response.pagerduty.com severity levels and incident roles, read 2026-11-10"
principles-of-chaos: "principlesofchaos.org, read 2026-11-10"
Reference Guide
Concepts
| Topic |
Reference |
When to Consult |
| SLI/SLO/SLA, error budget |
slo-sli-error-budget.md |
Setting targets, choosing SLIs per service type, budget arithmetic, the policy |
| Burn-rate alerting |
burn-rate-alerting.md |
Turning an SLO into pages and tickets, noisy or silent alerts, low traffic |
| Incident response |
incident-response.md |
Anything broken in production, roles, severity, communication, handoff |
| On-call and toil |
on-call-toil.md |
Rotation, paging rules, escalation, toil measurement |
| Postmortems |
postmortem.md |
After an incident, reviewing a draft, action items not closing |
| Change and rollout |
change-rollout.md |
Any production change, canary, freezes, rollback, patch windows |
| Runbooks |
runbooks.md |
Writing, testing, automating a procedure |
| Capacity planning |
capacity-planning.md |
Forecast, headroom, load testing, order dates |
| Production readiness |
production-readiness.md |
Before go-live or taking over a service |
| DORA metrics |
dora-metrics.md |
Measuring delivery performance with the current definitions |
| Troubleshooting |
troubleshooting-method.md |
Unknown cause: what changed, blast radius, hypothesis and test |
| Game days |
game-days.md |
Testing runbooks and failover, chaos experiments (pointer) |
Templates
| Template |
When to Use |
| slo-definition.md |
Writing the SLO document for a service |
| error-budget-policy.md |
Formalizing what happens when the budget is spent |
| burn-rate-alerts.rules.yml |
Prometheus recording and burn-rate alert rules (99.9% / 30 d), placeholders to replace |
| incident-checklist.md |
Live incident document with roles, timeline and close-out |
| postmortem.md |
Blameless postmortem |
| runbook.md |
Runbook for an alert or operation |
| change-request-rollout-plan.md |
Change with staged rollout and rollback criteria |
| prr-checklist.md |
Production readiness review |
| oncall-handoff.md |
End-of-shift handoff note |
Quick Reference
error budget = 1 - SLO 99.9% / 30 d -> 0.001 -> 43.2 min of full outage
burn rate = error_rate / (1 - SLO) alert threshold = burn_rate * (1 - SLO)
budget consumed = burn_rate * window / period 14.4 * 1h / 720h = 2% 6 * 6h / 720h = 5% 1 * 72h / 720h = 10%
time to exhaust = period / burn_rate 720h / 14.4 = 50h 720h / 6 = 5 d
max burn rate = 1 / (1 - SLO) 90% SLO -> 10, so a 14.4 page can never fire
promtool check rules references/templates/burn-rate-alerts.rules.yml # exit 0 valid, 1 error
Production action block (every rollback, failover, restart, traffic shift, freeze lift):
ACTION exact command with explicit target
EFFECT what changes, blast radius, what cannot be undone
PROOF check that it worked + check that nothing else broke
WAIT-FOR-GO who gives the go; stop here
Best Practices
DO
- Write the error budget policy before the first SLO miss and get it signed by the product owner
- Link a runbook in every paging alert's annotation; test runbooks in game days
- Use one alert parameter set for all services; bucket request classes instead of tuning each endpoint
- Record the budget consumed in every postmortem and link incidents to the deployments that caused them
- Keep incident documents and runbooks outside the system being repaired
DON'T
- Set a 100% target, or alert on a bare
for: duration of a short window
- Debug as the incident commander, or let several people change production at once
- Argue rollback criteria during the change, or roll back a cluster without reverting Git when GitOps owns it
- Report MTTR as a DORA metric, or set DORA metrics as team goals
- Duplicate sibling content: telemetry configuration is devops-observability, restore tests and DR are devops-storage-backup
Related skills
- devops-observability
- devops-storage-backup
- devops-cicd
- devops-kubernetes
- devops-gitops-packaging
- devops-proxmox
- devops-linux
- devops-secrets
1---2name: devops-sre-ops-23description: Use when defining or reviewing reliability targets and operations practice - SLI/SLO/SLA and error budgets (and the error budget policy), multi-window multi-burn-rate alerts, incident response (roles, severity, mitigate first, handoff), on-call rotation and toil, blameless postmortems, change management and safe rollout (canary, freeze, rollback criteria decided in advance), runbooks, capacity planning, production readiness reviews, DORA metrics, maintenance windows and patch cadence, game days, and structured troubleshooting of an unknown outage. Not for metrics/logs/tracing tooling (devops-observability), backup/restore and DR mechanics (devops-storage-backup), pipeline implementation (devops-cicd), security incident response (security-expert) or FinOps.4metadata:5 source: https://skillmd.com/skills/fusengine/devops-sre-ops-26---78<objective>9Covers the reliability-engineering and day-2 operations method for an IT/DevOps10agent: what to measure and promise (SLI, SLO, error budget and its policy), how11to alert on it (multi-window multi-burn-rate), how to respond (incidents,12on-call, troubleshooting), how to learn (postmortems, game days), how to change13production safely (rollout, rollback criteria, freezes, maintenance windows),14and how to prepare (runbooks, capacity, production readiness, toil, DORA).15It is method, not tooling. Telemetry stacks belong to devops-observability;16backup, restore and disaster recovery mechanics to devops-storage-backup;17pipelines to devops-cicd; security incidents to security-expert.18</objective>1920# DevOps SRE Operations2122## Agent Workflow (MANDATORY)2324Before applying this method to a real system, spawn in parallel, one `spawn_agent` call each:25261. **explore-codebase** - find existing SLO documents, alert rules, runbooks, postmortems, on-call and change records in the repo272. **research-expert** - verify any figure or tool option against the primary source (sre.google, dora.dev, prometheus.io) before relying on it283. **mcp__context7__query-docs** - only where a library or tool is involved (Prometheus rule syntax, Alertmanager); the method itself has no library2930Then read the live state read-only (current alerts, recent changes, budget remaining) before proposing anything.31After writing files in a repo, run **sniper**.3233## Sources and currency (checked 2026-11-10)3435Method statements trace to the Google SRE Book and SRE Workbook (online editions), DORA (dora.dev), PagerDuty's public incident response documentation, Prometheus documentation and principlesofchaos.org. Each reference ends with the URLs read and a "Not verified" list: figures that are Google-scale (on-call staffing, response times) are labeled as such; defaults proposed by this skill (update cadence, patch cadence, toil log length) are labeled as the skill's own.3637## Overview3839| Area | Decision in one line |40|------|----------------------|41| **SLO** | 100% is the wrong target; SLI = good events / total events; error budget = 1 - SLO; an SLO without a signed error budget policy has no teeth |42| **Alerting** | Page on budget burn: 14.4x over 1 h and 5 min, 6x over 6 h and 30 min; ticket on 1x over 3 d and 6 h (99.9% SLO, 30 days); short window = 1/12 of the long one |43| **Incidents** | Declare early; separate IC, ops, communications; mitigate before root cause; hand off command explicitly |44| **On-call** | Pages are urgent and actionable; at most about 2 incidents per shift; sustainable rotation; handoff in writing |45| **Postmortem** | Blameless, quantified impact, timeline in UTC, contributing factors, actions with type and one owner |46| **Change** | Small, staged, canary first; rollback criteria written before the change; budget exhausted = freeze except P0 and security |47| **Production actions** | Always ACTION / EFFECT / PROOF / WAIT-FOR-GO; the agent proposes, a human says go |48| **Toil** | Keep below 50% of time; measure, then design away, automate or document |49| **DORA** | Five metrics (lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate); MTTR is the old name |5051## Critical Rules52531. **Measure before promising** - an SLO needs a defined SLI, a measurement point, an owner and a policy ([slo-sli-error-budget.md](references/slo-sli-error-budget.md))542. **Alert on user-visible budget burn, with both windows** - never a bare `for:` on a short window ([burn-rate-alerting.md](references/burn-rate-alerting.md))553. **Mitigate first, then find the cause** - stop the bleeding, preserve evidence ([incident-response.md](references/incident-response.md), [troubleshooting-method.md](references/troubleshooting-method.md))564. **One change at a time, one commander** - only the operations lead changes production during an incident575. **Rollback criteria are decided before the change**, in numbers, in the change request ([change-rollout.md](references/change-rollout.md))586. **Every production action uses ACTION / EFFECT / PROOF / WAIT-FOR-GO** and waits for the go; the agent never rolls back, fails over, restarts, shifts traffic or lifts a freeze on its own initiative597. **Blameless, with owners** - postmortems fix systems, each action item has a type, an owner and a ticket ([postmortem.md](references/postmortem.md))608. **No invented figures** - a number not in a verified source is labeled as this skill's default or marked not verified; derive burn-rate and budget values with the formulas, show the arithmetic619. **Never print secrets** - runbooks and incident documents name the store path, not the value (devops-secrets)6263## Architecture6465```66reliability/67├── slos/<service>.md # filled slo-definition.md, one per service68├── policy/error-budget-policy.md # signed by the product owner69├── alerts/<service>.rules.yml # recording + burn-rate alert rules, promtool-checked70├── runbooks/<alert-or-operation>.md71├── incidents/<id>.md # live document, then the postmortem72├── changes/<chg-id>.md # change request and rollout plan73├── readiness/<service>-prr.md74└── oncall/handoff-<date>.md75```7677## Versions7879Pinned versions:8081```yaml82google-sre-book: "Site Reliability Engineering, O'Reilly 2016, web edition at sre.google/sre-book (page copyright 2017), read 2026-11-10"83google-sre-workbook: "The Site Reliability Workbook, O'Reilly 2018, web edition at sre.google/workbook, read 2026-11-10"84dora: "five-metric software delivery model; dora.dev guide updated 2026-01-05, history page updated 2026-01-02"85prometheus-rules: "rule file syntax from prometheus.io docs (latest), read 2026-11-10; promtool not run (not installed on authoring host)"86pagerduty-response-docs: "response.pagerduty.com severity levels and incident roles, read 2026-11-10"87principles-of-chaos: "principlesofchaos.org, read 2026-11-10"88```8990## Reference Guide9192### Concepts9394| Topic | Reference | When to Consult |95|-------|-----------|-----------------|96| **SLI/SLO/SLA, error budget** | [slo-sli-error-budget.md](references/slo-sli-error-budget.md) | Setting targets, choosing SLIs per service type, budget arithmetic, the policy |97| **Burn-rate alerting** | [burn-rate-alerting.md](references/burn-rate-alerting.md) | Turning an SLO into pages and tickets, noisy or silent alerts, low traffic |98| **Incident response** | [incident-response.md](references/incident-response.md) | Anything broken in production, roles, severity, communication, handoff |99| **On-call and toil** | [on-call-toil.md](references/on-call-toil.md) | Rotation, paging rules, escalation, toil measurement |100| **Postmortems** | [postmortem.md](references/postmortem.md) | After an incident, reviewing a draft, action items not closing |101| **Change and rollout** | [change-rollout.md](references/change-rollout.md) | Any production change, canary, freezes, rollback, patch windows |102| **Runbooks** | [runbooks.md](references/runbooks.md) | Writing, testing, automating a procedure |103| **Capacity planning** | [capacity-planning.md](references/capacity-planning.md) | Forecast, headroom, load testing, order dates |104| **Production readiness** | [production-readiness.md](references/production-readiness.md) | Before go-live or taking over a service |105| **DORA metrics** | [dora-metrics.md](references/dora-metrics.md) | Measuring delivery performance with the current definitions |106| **Troubleshooting** | [troubleshooting-method.md](references/troubleshooting-method.md) | Unknown cause: what changed, blast radius, hypothesis and test |107| **Game days** | [game-days.md](references/game-days.md) | Testing runbooks and failover, chaos experiments (pointer) |108109### Templates110111| Template | When to Use |112|----------|-------------|113| [slo-definition.md](references/templates/slo-definition.md) | Writing the SLO document for a service |114| [error-budget-policy.md](references/templates/error-budget-policy.md) | Formalizing what happens when the budget is spent |115| [burn-rate-alerts.rules.yml](references/templates/burn-rate-alerts.rules.yml) | Prometheus recording and burn-rate alert rules (99.9% / 30 d), placeholders to replace |116| [incident-checklist.md](references/templates/incident-checklist.md) | Live incident document with roles, timeline and close-out |117| [postmortem.md](references/templates/postmortem.md) | Blameless postmortem |118| [runbook.md](references/templates/runbook.md) | Runbook for an alert or operation |119| [change-request-rollout-plan.md](references/templates/change-request-rollout-plan.md) | Change with staged rollout and rollback criteria |120| [prr-checklist.md](references/templates/prr-checklist.md) | Production readiness review |121| [oncall-handoff.md](references/templates/oncall-handoff.md) | End-of-shift handoff note |122123## Quick Reference124125```text126error budget = 1 - SLO 99.9% / 30 d -> 0.001 -> 43.2 min of full outage127burn rate = error_rate / (1 - SLO) alert threshold = burn_rate * (1 - SLO)128budget consumed = burn_rate * window / period 14.4 * 1h / 720h = 2% 6 * 6h / 720h = 5% 1 * 72h / 720h = 10%129time to exhaust = period / burn_rate 720h / 14.4 = 50h 720h / 6 = 5 d130max burn rate = 1 / (1 - SLO) 90% SLO -> 10, so a 14.4 page can never fire131```132133```bash134promtool check rules references/templates/burn-rate-alerts.rules.yml # exit 0 valid, 1 error135```136137Production action block (every rollback, failover, restart, traffic shift, freeze lift):138139```text140ACTION exact command with explicit target141EFFECT what changes, blast radius, what cannot be undone142PROOF check that it worked + check that nothing else broke143WAIT-FOR-GO who gives the go; stop here144```145146## Best Practices147148### DO149- Write the error budget policy before the first SLO miss and get it signed by the product owner150- Link a runbook in every paging alert's annotation; test runbooks in game days151- Use one alert parameter set for all services; bucket request classes instead of tuning each endpoint152- Record the budget consumed in every postmortem and link incidents to the deployments that caused them153- Keep incident documents and runbooks outside the system being repaired154155### DON'T156- Set a 100% target, or alert on a bare `for:` duration of a short window157- Debug as the incident commander, or let several people change production at once158- Argue rollback criteria during the change, or roll back a cluster without reverting Git when GitOps owns it159- Report MTTR as a DORA metric, or set DORA metrics as team goals160- Duplicate sibling content: telemetry configuration is devops-observability, restore tests and DR are devops-storage-backup161162---163164## Related skills165166- devops-observability167- devops-storage-backup168- devops-cicd169- devops-kubernetes170- devops-gitops-packaging171- devops-proxmox172- devops-linux173- devops-secrets