DevOps SRE Operations
Agent Workflow (MANDATORY)
Before applying this method to a real system, spawn in parallel, one Agent call each with a name:
- fuse-ai-pilot:explore-codebase - find existing SLO documents, alert rules, runbooks, postmortems, on-call and change records in the repo
- fuse-ai-pilot:research-expert - verify any figure or tool option against the primary source (sre.google, dora.dev, prometheus.io) before relying on it
- mcp__context7__query-docs - only where a library or tool is involved (Prometheus rule syntax, Alertmanager); the method itself has no library
Then read the live state read-only (current alerts, recent changes, budget remaining) before proposing anything.
After writing files in a repo, run fuse-ai-pilot:sniper.
Sources and currency (checked 2026-11-10)
Method statements trace to the Google SRE Book and SRE Workbook (online editions), DORA (dora.dev), PagerDuty's public incident response documentation, Prometheus documentation and principlesofchaos.org. Each reference ends with the URLs read and a "Not verified" list: figures that are Google-scale (on-call staffing, response times) are labeled as such; defaults proposed by this skill (update cadence, patch cadence, toil log length) are labeled as the skill's own.
Overview
| Area |
Decision in one line |
| SLO |
100% is the wrong target; SLI = good events / total events; error budget = 1 - SLO; an SLO without a signed error budget policy has no teeth |
| Alerting |
Page on budget burn: 14.4x over 1 h and 5 min, 6x over 6 h and 30 min; ticket on 1x over 3 d and 6 h (99.9% SLO, 30 days); short window = 1/12 of the long one |
| Incidents |
Declare early; separate IC, ops, communications; mitigate before root cause; hand off command explicitly |
| On-call |
Pages are urgent and actionable; at most about 2 incidents per shift; sustainable rotation; handoff in writing |
| Postmortem |
Blameless, quantified impact, timeline in UTC, contributing factors, actions with type and one owner |
| Change |
Small, staged, canary first; rollback criteria written before the change; budget exhausted = freeze except P0 and security |
| Production actions |
Always ACTION / EFFECT / PROOF / WAIT-FOR-GO; the agent proposes, a human says go |
| Toil |
Keep below 50% of time; measure, then design away, automate or document |
| DORA |
Five metrics (lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate); MTTR is the old name |
Critical Rules
- Measure before promising - an SLO needs a defined SLI, a measurement point, an owner and a policy (slo-sli-error-budget.md)
- Alert on user-visible budget burn, with both windows - never a bare
for: on a short window (burn-rate-alerting.md)
- Mitigate first, then find the cause - stop the bleeding, preserve evidence (incident-response.md, troubleshooting-method.md)
- One change at a time, one commander - only the operations lead changes production during an incident
- Rollback criteria are decided before the change, in numbers, in the change request (change-rollout.md)
- Every production action uses ACTION / EFFECT / PROOF / WAIT-FOR-GO and waits for the go; the agent never rolls back, fails over, restarts, shifts traffic or lifts a freeze on its own initiative
- Blameless, with owners - postmortems fix systems, each action item has a type, an owner and a ticket (postmortem.md)
- No invented figures - a number not in a verified source is labeled as this skill's default or marked not verified; derive burn-rate and budget values with the formulas, show the arithmetic
- Never print secrets - runbooks and incident documents name the store path, not the value (devops-secrets)
Architecture
reliability/
├── slos/<service>.md # filled slo-definition.md, one per service
├── policy/error-budget-policy.md # signed by the product owner
├── alerts/<service>.rules.yml # recording + burn-rate alert rules, promtool-checked
├── runbooks/<alert-or-operation>.md
├── incidents/<id>.md # live document, then the postmortem
├── changes/<chg-id>.md # change request and rollout plan
├── readiness/<service>-prr.md
└── oncall/handoff-<date>.md
Reference Guide
Concepts
| Topic |
Reference |
When to Consult |
| SLI/SLO/SLA, error budget |
slo-sli-error-budget.md |
Setting targets, choosing SLIs per service type, budget arithmetic, the policy |
| Burn-rate alerting |
burn-rate-alerting.md |
Turning an SLO into pages and tickets, noisy or silent alerts, low traffic |
| Incident response |
incident-response.md |
Anything broken in production, roles, severity, communication, handoff |
| On-call and toil |
on-call-toil.md |
Rotation, paging rules, escalation, toil measurement |
| Postmortems |
postmortem.md |
After an incident, reviewing a draft, action items not closing |
| Change and rollout |
change-rollout.md |
Any production change, canary, freezes, rollback, patch windows |
| Runbooks |
runbooks.md |
Writing, testing, automating a procedure |
| Capacity planning |
capacity-planning.md |
Forecast, headroom, load testing, order dates |
| Production readiness |
production-readiness.md |
Before go-live or taking over a service |
| DORA metrics |
dora-metrics.md |
Measuring delivery performance with the current definitions |
| Troubleshooting |
troubleshooting-method.md |
Unknown cause: what changed, blast radius, hypothesis and test |
| Game days |
game-days.md |
Testing runbooks and failover, chaos experiments (pointer) |
Templates
| Template |
When to Use |
| slo-definition.md |
Writing the SLO document for a service |
| error-budget-policy.md |
Formalizing what happens when the budget is spent |
| burn-rate-alerts.rules.yml |
Prometheus recording and burn-rate alert rules (99.9% / 30 d), placeholders to replace |
| incident-checklist.md |
Live incident document with roles, timeline and close-out |
| postmortem.md |
Blameless postmortem |
| runbook.md |
Runbook for an alert or operation |
| change-request-rollout-plan.md |
Change with staged rollout and rollback criteria |
| prr-checklist.md |
Production readiness review |
| oncall-handoff.md |
End-of-shift handoff note |
Quick Reference
error budget = 1 - SLO 99.9% / 30 d -> 0.001 -> 43.2 min of full outage
burn rate = error_rate / (1 - SLO) alert threshold = burn_rate * (1 - SLO)
budget consumed = burn_rate * window / period 14.4 * 1h / 720h = 2% 6 * 6h / 720h = 5% 1 * 72h / 720h = 10%
time to exhaust = period / burn_rate 720h / 14.4 = 50h 720h / 6 = 5 d
max burn rate = 1 / (1 - SLO) 90% SLO -> 10, so a 14.4 page can never fire
promtool check rules references/templates/burn-rate-alerts.rules.yml # exit 0 valid, 1 error
Production action block (every rollback, failover, restart, traffic shift, freeze lift):
ACTION exact command with explicit target
EFFECT what changes, blast radius, what cannot be undone
PROOF check that it worked + check that nothing else broke
WAIT-FOR-GO who gives the go; stop here
Best Practices
DO
- Write the error budget policy before the first SLO miss and get it signed by the product owner
- Link a runbook in every paging alert's annotation; test runbooks in game days
- Use one alert parameter set for all services; bucket request classes instead of tuning each endpoint
- Record the budget consumed in every postmortem and link incidents to the deployments that caused them
- Keep incident documents and runbooks outside the system being repaired
DON'T
- Set a 100% target, or alert on a bare
for: duration of a short window
- Debug as the incident commander, or let several people change production at once
- Argue rollback criteria during the change, or roll back a cluster without reverting Git when GitOps owns it
- Report MTTR as a DORA metric, or set DORA metrics as team goals
- Duplicate sibling content: telemetry configuration is devops-observability, restore tests and DR are devops-storage-backup
1---2name: devops-sre-ops3description: Use when defining or reviewing reliability targets and operations practice - SLI/SLO/SLA and error budgets (and the error budget policy), multi-window multi-burn-rate alerts, incident response (roles, severity, mitigate first, handoff), on-call rotation and toil, blameless postmortems, change management and safe rollout (canary, freeze, rollback criteria decided in advance), runbooks, capacity planning, production readiness reviews, DORA metrics, maintenance windows and patch cadence, game days, and structured troubleshooting of an unknown outage. Not for metrics/logs/tracing tooling (devops-observability), backup/restore and DR mechanics (devops-storage-backup), pipeline implementation (devops-cicd), security incident response (fuse-security:security-expert) or FinOps.4versions:5 google-sre-book: "Site Reliability Engineering, O'Reilly 2016, web edition at sre.google/sre-book (page copyright 2017), read 2026-11-10"6 google-sre-workbook: "The Site Reliability Workbook, O'Reilly 2018, web edition at sre.google/workbook, read 2026-11-10"7 dora: "five-metric software delivery model; dora.dev guide updated 2026-01-05, history page updated 2026-01-02"8 prometheus-rules: "rule file syntax from prometheus.io docs (latest), read 2026-11-10; promtool not run (not installed on authoring host)"9 pagerduty-response-docs: "response.pagerduty.com severity levels and incident roles, read 2026-11-10"10 principles-of-chaos: "principlesofchaos.org, read 2026-11-10"11user-invocable: true12references:13 - references/slo-sli-error-budget.md14 - references/burn-rate-alerting.md15 - references/incident-response.md16 - references/on-call-toil.md17 - references/postmortem.md18 - references/change-rollout.md19 - references/runbooks.md20 - references/capacity-planning.md21 - references/production-readiness.md22 - references/dora-metrics.md23 - references/troubleshooting-method.md24 - references/game-days.md25 - references/templates/slo-definition.md26 - references/templates/error-budget-policy.md27 - references/templates/burn-rate-alerts.rules.yml28 - references/templates/incident-checklist.md29 - references/templates/postmortem.md30 - references/templates/runbook.md31 - references/templates/change-request-rollout-plan.md32 - references/templates/prr-checklist.md33 - references/templates/oncall-handoff.md34related-skills:35 - devops-observability36 - devops-storage-backup37 - devops-cicd38 - devops-kubernetes39 - devops-gitops-packaging40 - devops-proxmox41 - devops-linux42 - devops-secrets43metadata:44 source: https://skillmd.com/skills/fusengine/devops-sre-ops45---4647<objective>48Covers the reliability-engineering and day-2 operations method for an IT/DevOps49agent: what to measure and promise (SLI, SLO, error budget and its policy), how50to alert on it (multi-window multi-burn-rate), how to respond (incidents,51on-call, troubleshooting), how to learn (postmortems, game days), how to change52production safely (rollout, rollback criteria, freezes, maintenance windows),53and how to prepare (runbooks, capacity, production readiness, toil, DORA).54It is method, not tooling. Telemetry stacks belong to devops-observability;55backup, restore and disaster recovery mechanics to devops-storage-backup;56pipelines to devops-cicd; security incidents to fuse-security:security-expert.57</objective>5859# DevOps SRE Operations6061## Agent Workflow (MANDATORY)6263Before applying this method to a real system, spawn in parallel, one `Agent` call each with a `name`:64651. **fuse-ai-pilot:explore-codebase** - find existing SLO documents, alert rules, runbooks, postmortems, on-call and change records in the repo662. **fuse-ai-pilot:research-expert** - verify any figure or tool option against the primary source (sre.google, dora.dev, prometheus.io) before relying on it673. **mcp__context7__query-docs** - only where a library or tool is involved (Prometheus rule syntax, Alertmanager); the method itself has no library6869Then read the live state read-only (current alerts, recent changes, budget remaining) before proposing anything.70After writing files in a repo, run **fuse-ai-pilot:sniper**.7172## Sources and currency (checked 2026-11-10)7374Method statements trace to the Google SRE Book and SRE Workbook (online editions), DORA (dora.dev), PagerDuty's public incident response documentation, Prometheus documentation and principlesofchaos.org. Each reference ends with the URLs read and a "Not verified" list: figures that are Google-scale (on-call staffing, response times) are labeled as such; defaults proposed by this skill (update cadence, patch cadence, toil log length) are labeled as the skill's own.7576## Overview7778| Area | Decision in one line |79|------|----------------------|80| **SLO** | 100% is the wrong target; SLI = good events / total events; error budget = 1 - SLO; an SLO without a signed error budget policy has no teeth |81| **Alerting** | Page on budget burn: 14.4x over 1 h and 5 min, 6x over 6 h and 30 min; ticket on 1x over 3 d and 6 h (99.9% SLO, 30 days); short window = 1/12 of the long one |82| **Incidents** | Declare early; separate IC, ops, communications; mitigate before root cause; hand off command explicitly |83| **On-call** | Pages are urgent and actionable; at most about 2 incidents per shift; sustainable rotation; handoff in writing |84| **Postmortem** | Blameless, quantified impact, timeline in UTC, contributing factors, actions with type and one owner |85| **Change** | Small, staged, canary first; rollback criteria written before the change; budget exhausted = freeze except P0 and security |86| **Production actions** | Always ACTION / EFFECT / PROOF / WAIT-FOR-GO; the agent proposes, a human says go |87| **Toil** | Keep below 50% of time; measure, then design away, automate or document |88| **DORA** | Five metrics (lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate); MTTR is the old name |8990## Critical Rules91921. **Measure before promising** - an SLO needs a defined SLI, a measurement point, an owner and a policy ([slo-sli-error-budget.md](references/slo-sli-error-budget.md))932. **Alert on user-visible budget burn, with both windows** - never a bare `for:` on a short window ([burn-rate-alerting.md](references/burn-rate-alerting.md))943. **Mitigate first, then find the cause** - stop the bleeding, preserve evidence ([incident-response.md](references/incident-response.md), [troubleshooting-method.md](references/troubleshooting-method.md))954. **One change at a time, one commander** - only the operations lead changes production during an incident965. **Rollback criteria are decided before the change**, in numbers, in the change request ([change-rollout.md](references/change-rollout.md))976. **Every production action uses ACTION / EFFECT / PROOF / WAIT-FOR-GO** and waits for the go; the agent never rolls back, fails over, restarts, shifts traffic or lifts a freeze on its own initiative987. **Blameless, with owners** - postmortems fix systems, each action item has a type, an owner and a ticket ([postmortem.md](references/postmortem.md))998. **No invented figures** - a number not in a verified source is labeled as this skill's default or marked not verified; derive burn-rate and budget values with the formulas, show the arithmetic1009. **Never print secrets** - runbooks and incident documents name the store path, not the value (devops-secrets)101102## Architecture103104```105reliability/106├── slos/<service>.md # filled slo-definition.md, one per service107├── policy/error-budget-policy.md # signed by the product owner108├── alerts/<service>.rules.yml # recording + burn-rate alert rules, promtool-checked109├── runbooks/<alert-or-operation>.md110├── incidents/<id>.md # live document, then the postmortem111├── changes/<chg-id>.md # change request and rollout plan112├── readiness/<service>-prr.md113└── oncall/handoff-<date>.md114```115116## Reference Guide117118### Concepts119120| Topic | Reference | When to Consult |121|-------|-----------|-----------------|122| **SLI/SLO/SLA, error budget** | [slo-sli-error-budget.md](references/slo-sli-error-budget.md) | Setting targets, choosing SLIs per service type, budget arithmetic, the policy |123| **Burn-rate alerting** | [burn-rate-alerting.md](references/burn-rate-alerting.md) | Turning an SLO into pages and tickets, noisy or silent alerts, low traffic |124| **Incident response** | [incident-response.md](references/incident-response.md) | Anything broken in production, roles, severity, communication, handoff |125| **On-call and toil** | [on-call-toil.md](references/on-call-toil.md) | Rotation, paging rules, escalation, toil measurement |126| **Postmortems** | [postmortem.md](references/postmortem.md) | After an incident, reviewing a draft, action items not closing |127| **Change and rollout** | [change-rollout.md](references/change-rollout.md) | Any production change, canary, freezes, rollback, patch windows |128| **Runbooks** | [runbooks.md](references/runbooks.md) | Writing, testing, automating a procedure |129| **Capacity planning** | [capacity-planning.md](references/capacity-planning.md) | Forecast, headroom, load testing, order dates |130| **Production readiness** | [production-readiness.md](references/production-readiness.md) | Before go-live or taking over a service |131| **DORA metrics** | [dora-metrics.md](references/dora-metrics.md) | Measuring delivery performance with the current definitions |132| **Troubleshooting** | [troubleshooting-method.md](references/troubleshooting-method.md) | Unknown cause: what changed, blast radius, hypothesis and test |133| **Game days** | [game-days.md](references/game-days.md) | Testing runbooks and failover, chaos experiments (pointer) |134135### Templates136137| Template | When to Use |138|----------|-------------|139| [slo-definition.md](references/templates/slo-definition.md) | Writing the SLO document for a service |140| [error-budget-policy.md](references/templates/error-budget-policy.md) | Formalizing what happens when the budget is spent |141| [burn-rate-alerts.rules.yml](references/templates/burn-rate-alerts.rules.yml) | Prometheus recording and burn-rate alert rules (99.9% / 30 d), placeholders to replace |142| [incident-checklist.md](references/templates/incident-checklist.md) | Live incident document with roles, timeline and close-out |143| [postmortem.md](references/templates/postmortem.md) | Blameless postmortem |144| [runbook.md](references/templates/runbook.md) | Runbook for an alert or operation |145| [change-request-rollout-plan.md](references/templates/change-request-rollout-plan.md) | Change with staged rollout and rollback criteria |146| [prr-checklist.md](references/templates/prr-checklist.md) | Production readiness review |147| [oncall-handoff.md](references/templates/oncall-handoff.md) | End-of-shift handoff note |148149## Quick Reference150151```text152error budget = 1 - SLO 99.9% / 30 d -> 0.001 -> 43.2 min of full outage153burn rate = error_rate / (1 - SLO) alert threshold = burn_rate * (1 - SLO)154budget consumed = burn_rate * window / period 14.4 * 1h / 720h = 2% 6 * 6h / 720h = 5% 1 * 72h / 720h = 10%155time to exhaust = period / burn_rate 720h / 14.4 = 50h 720h / 6 = 5 d156max burn rate = 1 / (1 - SLO) 90% SLO -> 10, so a 14.4 page can never fire157```158159```bash160promtool check rules references/templates/burn-rate-alerts.rules.yml # exit 0 valid, 1 error161```162163Production action block (every rollback, failover, restart, traffic shift, freeze lift):164165```text166ACTION exact command with explicit target167EFFECT what changes, blast radius, what cannot be undone168PROOF check that it worked + check that nothing else broke169WAIT-FOR-GO who gives the go; stop here170```171172## Best Practices173174### DO175- Write the error budget policy before the first SLO miss and get it signed by the product owner176- Link a runbook in every paging alert's annotation; test runbooks in game days177- Use one alert parameter set for all services; bucket request classes instead of tuning each endpoint178- Record the budget consumed in every postmortem and link incidents to the deployments that caused them179- Keep incident documents and runbooks outside the system being repaired180181### DON'T182- Set a 100% target, or alert on a bare `for:` duration of a short window183- Debug as the incident commander, or let several people change production at once184- Argue rollback criteria during the change, or roll back a cluster without reverting Git when GitOps owns it185- Report MTTR as a DORA metric, or set DORA metrics as team goals186- Duplicate sibling content: telemetry configuration is devops-observability, restore tests and DR are devops-storage-backup