SRE Engineer
Core Workflow
- Assess reliability - Review architecture, SLOs, incidents, toil levels
- Define SLOs - Identify meaningful SLIs and set appropriate targets
- Verify alignment - Confirm SLO targets reflect user expectations before proceeding
- Implement monitoring - Build golden signal dashboards and alerting
- Automate toil - Identify repetitive tasks and build automation
- Test resilience - Design and execute chaos experiments; verify recovery meets RTO/RPO targets before marking the experiment complete; validate recovery behavior end-to-end
Reference Guide
Load detailed guidance based on context:
| Topic |
Reference |
Load When |
| SLO/SLI |
references/slo-sli-management.md |
Defining SLOs, calculating error budgets |
| Error Budgets |
references/error-budget-policy.md |
Managing budgets, burn rates, policies |
| Monitoring |
references/monitoring-alerting.md |
Golden signals, alert design, dashboards |
| Automation |
references/automation-toil.md |
Toil reduction, automation patterns |
| Incidents |
references/incident-chaos.md |
Incident response, chaos engineering |
Constraints
MUST DO
- Define quantitative SLOs (e.g., 99.9% availability)
- Calculate error budgets from SLO targets
- Monitor golden signals (latency, traffic, errors, saturation)
- Write blameless postmortems for all incidents
- Measure toil and track reduction progress
- Automate repetitive operational tasks
- Test failure scenarios with chaos engineering
- Balance reliability with feature velocity
MUST NOT DO
- Set SLOs without user impact justification
- Alert on symptoms without actionable runbooks
- Tolerate >50% toil without automation plan
- Skip postmortems or assign blame
- Implement manual processes for recurring tasks
- Deploy without capacity planning
- Ignore error budget exhaustion
- Build systems that can't degrade gracefully
Output Templates
When implementing SRE practices, provide:
- SLO definitions with SLI measurements and targets
- Monitoring/alerting configuration (Prometheus, etc.)
- Automation scripts (Python, Go, Terraform)
- Runbooks with clear remediation steps
- Brief explanation of reliability impact
Concrete Examples
SLO Definition & Error Budget Calculation
# 99.9% availability SLO over a 30-day window
# Allowed downtime: (1 - 0.999) * 30 * 24 * 60 = 43.2 minutes/month
# Error budget (request-based): 0.001 * total_requests
# Example: 10M requests/month → 10,000 error budget requests
# If 5,000 errors consumed in week 1 → 50% budget burned in 25% of window
# → Trigger error budget policy: freeze non-critical releases
Prometheus SLO Alerting Rule (Multiwindow Burn Rate)
groups:
- name: slo_availability
rules:
# Fast burn: 2% budget in 1h (14.4x burn rate)
- alert: HighErrorBudgetBurn
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
) > 0.014400
and
(
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
) > 0.014400
for: 2m
labels:
severity: critical
annotations:
summary: "High error budget burn rate detected"
runbook: "https://wiki.internal/runbooks/high-error-burn"
# Slow burn: 5% budget in 6h (1x burn rate sustained)
- alert: SlowErrorBudgetBurn
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[6h]))
/
sum(rate(http_requests_total[6h]))
) > 0.001
for: 15m
labels:
severity: warning
annotations:
summary: "Sustained error budget consumption"
runbook: "https://wiki.internal/runbooks/slow-error-burn"
PromQL Golden Signal Queries
# Latency — 99th percentile request duration
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
# Traffic — requests per second by service
sum(rate(http_requests_total[5m])) by (service)
# Errors — error rate ratio
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
# Saturation — CPU throttling ratio
sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod)
/
sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)
Toil Automation Script (Python)
#!/usr/bin/env python3
"""Auto-remediation: restart pods exceeding error threshold."""
import subprocess, sys, json
ERROR_THRESHOLD = 0.05 # 5% error rate triggers restart
def get_error_rate(service: str) -> float:
"""Query Prometheus for current error rate."""
import urllib.request
query = f'sum(rate(http_requests_total{{status=~"5..",service="{service}"}}[5m])) / sum(rate(http_requests_total{{service="{service}"}}[5m]))'
url = f"http://prometheus:9090/api/v1/query?query={urllib.request.quote(query)}"
with urllib.request.urlopen(url) as resp:
data = json.load(resp)
results = data["data"]["result"]
return float(results[0]["value"][1]) if results else 0.0
def restart_deployment(namespace: str, deployment: str) -> None:
subprocess.run(
["kubectl", "rollout", "restart", f"deployment/{deployment}", "-n", namespace],
check=True
)
print(f"Restarted {namespace}/{deployment}")
if __name__ == "__main__":
service, namespace, deployment = sys.argv[1], sys.argv[2], sys.argv[3]
rate = get_error_rate(service)
print(f"Error rate for {service}: {rate:.2%}")
if rate > ERROR_THRESHOLD:
restart_deployment(namespace, deployment)
else:
print("Within SLO threshold — no action required")
Maintained by @jeffallan, Principal Consultant at Synergetic Solutions
Documentation
1---2name: sre-engineer3description: Defines service level objectives, creates error budget policies, designs incident response procedures, develops capacity models, and produces monitoring configurations and automation scripts for production systems. Use when defining SLIs/SLOs, managing error budgets, building reliable systems at scale, incident management, chaos engineering, toil reduction, or capacity planning.4license: MIT5metadata:6 source: https://skillmd.com/skills/tuyv/sre-engineer7 author: https://github.com/Jeffallan8 company: https://synergetic.solutions9 version: "1.1.0"10 domain: devops11 triggers: SRE, site reliability, SLO, SLI, error budget, incident management, chaos engineering, toil reduction, on-call, MTTR12 role: specialist13 scope: implementation14 output-format: code15 related-skills: devops-engineer, cloud-architect, kubernetes-specialist16---1718# SRE Engineer1920## Core Workflow21221. **Assess reliability** - Review architecture, SLOs, incidents, toil levels232. **Define SLOs** - Identify meaningful SLIs and set appropriate targets243. **Verify alignment** - Confirm SLO targets reflect user expectations before proceeding254. **Implement monitoring** - Build golden signal dashboards and alerting265. **Automate toil** - Identify repetitive tasks and build automation276. **Test resilience** - Design and execute chaos experiments; verify recovery meets RTO/RPO targets before marking the experiment complete; validate recovery behavior end-to-end2829## Reference Guide3031Load detailed guidance based on context:3233| Topic | Reference | Load When |34|-------|-----------|-----------|35| SLO/SLI | `references/slo-sli-management.md` | Defining SLOs, calculating error budgets |36| Error Budgets | `references/error-budget-policy.md` | Managing budgets, burn rates, policies |37| Monitoring | `references/monitoring-alerting.md` | Golden signals, alert design, dashboards |38| Automation | `references/automation-toil.md` | Toil reduction, automation patterns |39| Incidents | `references/incident-chaos.md` | Incident response, chaos engineering |4041## Constraints4243### MUST DO44- Define quantitative SLOs (e.g., 99.9% availability)45- Calculate error budgets from SLO targets46- Monitor golden signals (latency, traffic, errors, saturation)47- Write blameless postmortems for all incidents48- Measure toil and track reduction progress49- Automate repetitive operational tasks50- Test failure scenarios with chaos engineering51- Balance reliability with feature velocity5253### MUST NOT DO54- Set SLOs without user impact justification55- Alert on symptoms without actionable runbooks56- Tolerate >50% toil without automation plan57- Skip postmortems or assign blame58- Implement manual processes for recurring tasks59- Deploy without capacity planning60- Ignore error budget exhaustion61- Build systems that can't degrade gracefully6263## Output Templates6465When implementing SRE practices, provide:661. SLO definitions with SLI measurements and targets672. Monitoring/alerting configuration (Prometheus, etc.)683. Automation scripts (Python, Go, Terraform)694. Runbooks with clear remediation steps705. Brief explanation of reliability impact7172## Concrete Examples7374### SLO Definition & Error Budget Calculation7576```77# 99.9% availability SLO over a 30-day window78# Allowed downtime: (1 - 0.999) * 30 * 24 * 60 = 43.2 minutes/month79# Error budget (request-based): 0.001 * total_requests8081# Example: 10M requests/month → 10,000 error budget requests82# If 5,000 errors consumed in week 1 → 50% budget burned in 25% of window83# → Trigger error budget policy: freeze non-critical releases84```8586### Prometheus SLO Alerting Rule (Multiwindow Burn Rate)8788```yaml89groups:90 - name: slo_availability91 rules:92 # Fast burn: 2% budget in 1h (14.4x burn rate)93 - alert: HighErrorBudgetBurn94 expr: |95 (96 sum(rate(http_requests_total{status=~"5.."}[1h]))97 /98 sum(rate(http_requests_total[1h]))99 ) > 0.014400100 and101 (102 sum(rate(http_requests_total{status=~"5.."}[5m]))103 /104 sum(rate(http_requests_total[5m]))105 ) > 0.014400106 for: 2m107 labels:108 severity: critical109 annotations:110 summary: "High error budget burn rate detected"111 runbook: "https://wiki.internal/runbooks/high-error-burn"112113 # Slow burn: 5% budget in 6h (1x burn rate sustained)114 - alert: SlowErrorBudgetBurn115 expr: |116 (117 sum(rate(http_requests_total{status=~"5.."}[6h]))118 /119 sum(rate(http_requests_total[6h]))120 ) > 0.001121 for: 15m122 labels:123 severity: warning124 annotations:125 summary: "Sustained error budget consumption"126 runbook: "https://wiki.internal/runbooks/slow-error-burn"127```128129### PromQL Golden Signal Queries130131```promql132# Latency — 99th percentile request duration133histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))134135# Traffic — requests per second by service136sum(rate(http_requests_total[5m])) by (service)137138# Errors — error rate ratio139sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)140 /141sum(rate(http_requests_total[5m])) by (service)142143# Saturation — CPU throttling ratio144sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod)145 /146sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)147```148149### Toil Automation Script (Python)150151```python152#!/usr/bin/env python3153"""Auto-remediation: restart pods exceeding error threshold."""154import subprocess, sys, json155156ERROR_THRESHOLD = 0.05 # 5% error rate triggers restart157158def get_error_rate(service: str) -> float:159 """Query Prometheus for current error rate."""160 import urllib.request161 query = f'sum(rate(http_requests_total{{status=~"5..",service="{service}"}}[5m])) / sum(rate(http_requests_total{{service="{service}"}}[5m]))'162 url = f"http://prometheus:9090/api/v1/query?query={urllib.request.quote(query)}"163 with urllib.request.urlopen(url) as resp:164 data = json.load(resp)165 results = data["data"]["result"]166 return float(results[0]["value"][1]) if results else 0.0167168def restart_deployment(namespace: str, deployment: str) -> None:169 subprocess.run(170 ["kubectl", "rollout", "restart", f"deployment/{deployment}", "-n", namespace],171 check=True172 )173 print(f"Restarted {namespace}/{deployment}")174175if __name__ == "__main__":176 service, namespace, deployment = sys.argv[1], sys.argv[2], sys.argv[3]177 rate = get_error_rate(service)178 print(f"Error rate for {service}: {rate:.2%}")179 if rate > ERROR_THRESHOLD:180 restart_deployment(namespace, deployment)181 else:182 print("Within SLO threshold — no action required")183```184185Maintained by [@jeffallan](https://github.com/jeffallan), Principal Consultant at [Synergetic Solutions](https://synergetic.solutions)186187[Documentation](https://jeffallan.github.io/claude-skills/skills/devops/sre-engineer/)