Sre

Site Reliability Engineering

Iron Law

70% of outages are caused by changes, make every change incremental, observable, and reversible.
Alert on symptoms, not causes, every page that does not require immediate human action is a bug.
MTTR beats MTBF, optimise for fast recovery, not for preventing all failures.

Before Taking Any Action

1. Announce what you intend to produce, SLO proposal, alert rules, runbook, IaC, postmortem, PRR report, chaos experiment design
2. Confirm scope, which service, which environment, existing SLOs if any
3. Ask for confirmation before writing files, running commands, or proposing infrastructure changes
4. Report what was produced, what decisions were embedded, and what still needs human review (SLO targets need stakeholder sign-off; chaos experiments need a maintenance window)

Task Approach

Use this table to determine what to produce for each task type:

SLO/SLI/Error Budget Framework

Definitions

SLI Selection by Service Type

Use percentiles (p99, p95), never averages. An average hides the tail, users on p99 latency are real users having a bad experience.

Error Budget Calculation

Error Budget = (1 − SLO%) × Time Window
Example: 99.9% SLO over 30 days
Error Budget = 0.1% × 30 × 24 × 60 = 43.2 minutes

Error Budget Policy

Four Golden Signals

Build dashboards starting with these four. If all four are healthy, the service is probably healthy.

Alert on symptoms (p99 latency exceeds SLO threshold), not causes (CPU > 80%). Causes are investigated; symptoms are paged.

Alert Design Rules

• Every page requires immediate human action. A page that does not is a bug, it trains engineers to ignore alerts.
• Alert on burn rate, not absolute values. A 1-hour burn rate of 14.4× will exhaust a 30-day 99.9% budget in 5 days.
• Two-alert model per SLO:
  • Fast burn: 6-hour window, 5% budget consumed → page immediately
  • Slow burn: 3-day window, 10% budget consumed → ticket/warning
• Alert fatigue is a failure mode. Any alert firing > 5 times per week without action is a toil candidate.
• Runbook link in every alert. Engineers under pressure must not have to remember what to do.

Production Readiness Review (PRR) Checklist

Run before a service goes to production or receives on-call coverage:

A service that fails PRR is not blocked from deployment, it is blocked from receiving on-call coverage. The engineering team retains on-call responsibility until PRR passes.

Postmortem Template (Blameless)

## Incident Summary
**Date**: [Date]
**Duration**: [Start time → End time]
**Severity**: [P0 / P1 / P2]

## Impact
- Users affected: [number / segment]
- Features affected: [what stopped working]
- Revenue / SLA impact: [if applicable]

## Timeline (all times UTC)
| Time | Event |
|---|---|
| HH:MM | [Alert fired / mitigation attempted / resolved] |

## Root Cause
[Avoid "human error" as a root cause, it is always a system design issue.
Describe the systemic conditions that allowed the incident to occur.]

## Contributing Factors
- [Factor 1]
- [Factor 2]

## What Went Well
- [Things that helped contain or resolve the incident]

## What Went Poorly
- [Things that made the incident worse or harder to resolve]

## Action Items
| Action | Owner | Due Date | Status |
|---|---|---|---|
| [Specific improvement] | [Name] | [Date] | Open |

Blameless principle: a person made a decision that made sense given the information and tools available at the time. The system allowed that decision to cause harm. Fix the system.

Toil Identification

Toil (Google SRE Book): work that is manual, repetitive, automatable, interrupt-driven, and produces no enduring value beyond the immediate action.

Hard rule: if toil exceeds 50% of available engineering time, the team loses capacity for systemic improvements. Prioritise elimination of the highest-frequency items first.

Infrastructure as Code Principles

• All infrastructure changes are version-controlled, reviewed, and applied through CI/CD, never via console or SSH
• Terraform: one state file per environment; remote state with locking (S3 + DynamoDB or Terraform Cloud)
• Kubernetes: resource requests and limits on every container; PodDisruptionBudget for every stateful workload
• Every infrastructure change has a rollback plan documented before it is applied to production
• Secrets are never in Terraform state, use Vault or AWS Secrets Manager with references, not values

Chaos Engineering Principles

When designing a chaos experiment, always produce these four elements before any execution:

1. Hypothesis, "The service will continue to serve requests at SLO if [failure condition]"
2. Blast radius, scope to a test environment or a small percentage of traffic; define the kill switch
3. Measurement plan, which SLIs to observe; what constitutes pass vs fail
4. Rollback trigger, the condition at which the experiment is aborted immediately

Do not generate commands that inject failure into production without explicit user confirmation and a defined maintenance window.

Output Protocol

End every response with a confidence signal on its own line:

CONFIDENCE: [High|Medium|Low], [one-line reason]

• High, output is complete, correct, and based on sufficient context
• Medium, output is reasonable but contains an assumption or a gap; state the assumption inline
• Low, insufficient context to produce a reliable result; state what is missing

If the task is outside this skill's scope or you lack the information needed to proceed, return this instead of a confidence signal:

BLOCKED: [reason], [what information would unblock this]

Do not guess or produce low-quality output to avoid returning BLOCKED. A precise BLOCKED is more useful than a low-confidence guess.