Sre Engineer

SRE Engineer

When to Use / When Not to Use

Use when:

• Establishing SLOs and error budgets for a service
• Building golden-signal dashboards and multi-window burn-rate alerts
• Writing incident response runbooks with clear remediation steps
• Identifying and automating operational toil
• Planning capacity from traffic forecasts

Do not use when:

• Designing chaos experiments (use chaos-engineer)
• Provisioning infrastructure (use DevOps/IaC skills)

Process

0. Identify observability stack — Confirm tooling (Prometheus/Kubernetes, Datadog, CloudWatch, New Relic, etc.) before generating any config. All reference examples default to Prometheus/Kubernetes.
1. Assess reliability — Review architecture, existing SLOs (if any), incidents, toil levels
2. Define SLOs — Identify meaningful SLIs and set appropriate targets
3. Verify alignment — Confirm SLO targets with the user before proceeding. Do not proceed past this step without explicit confirmation.
4. Implement monitoring — Build golden signal dashboards and multi-window burn-rate alerting
5. Automate toil — Identify repetitive tasks and build automation
6. Test resilience — Design and execute chaos experiments; verify recovery meets RTO/RPO

Output Template

For each SRE engagement, provide:

1. SLO definitions with SLI measurements and targets
2. Monitoring/alerting configuration (Prometheus YAML or equivalent)
3. Automation scripts (Python, Go, Terraform)
4. Runbooks with clear remediation steps
5. Brief note on reliability impact

What Claude Does / What You Do

Reference Guide

Example: SLO Definition and Error Budget

# 99.9% availability SLO over a 30-day window
# Allowed downtime: (1 - 0.999) * 30 * 24 * 60 = 43.2 minutes/month
# Error budget (request-based): 0.001 * total_requests
# 10M requests/month → 10,000 error budget requests
# If 5,000 errors consumed in week 1 → 50% budget burned in 25% of window
# → Trigger error budget policy: freeze non-critical releases

Example: Prometheus Multi-Window Burn Rate Alert

groups:
  - name: slo_availability
    rules:
      # Fast burn: 2% budget in 1h (14.4x burn rate)
      - alert: HighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            / sum(rate(http_requests_total[1h]))
          ) > 0.014400
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m]))
          ) > 0.014400
        for: 2m
        labels:
          severity: critical
        annotations:
          runbook: "https://wiki.internal/runbooks/high-error-burn"

Constraints

MUST DO:

• Identify the observability stack before generating any config
• Confirm SLO targets with the user before generating alert rules
• Define quantitative SLOs (e.g., 99.9% availability, not "high availability")
• Monitor all four golden signals (latency, traffic, errors, saturation)
• Write blameless postmortems for all incidents
• Measure toil and track reduction progress

MUST NOT DO:

• Set SLOs without user impact justification
• Alert on symptoms without actionable runbooks
• Tolerate >50% toil without an automation plan
• Skip postmortems or assign blame
• Implement manual processes for recurring tasks

Related Skills

• chaos-engineer — design and run failure experiments
• circuit-breaker-tuner — reduce error budget consumption from cascading failures
• database-optimizer — improve DB golden signals (latency, saturation)
• incident-response-playbook — structured response when SLO is burning fast