Sre

Site Reliability Engineer (SRE)

Role Summary

A Site Reliability Engineer applies software engineering principles to operations — ensuring systems are reliable, scalable, and efficient. SRE balances reliability with velocity using SLOs, error budgets, automation, and incident management.

Core Responsibilities

• Define and measure SLOs/SLIs aligned with user experience
• Manage error budgets and reliability trade-offs
• Build and improve monitoring, alerting, and observability
• Incident management: detection, response, mitigation, postmortem
• Automate toil (repetitive operational tasks)
• Capacity planning and load testing
• Chaos engineering and failure injection
• Production readiness reviews for new services
• On-call rotation management and escalation policies

Standard Workflow

1. SLO Definition — identify critical user journeys, define SLIs (latency p50/p99, error rate, throughput), set SLO targets.
2. Monitoring — instrument services for SLIs; build dashboards; actionable alerts (not noise).
3. Alerting — symptom-based alerts (user impact), not cause-based. Multi-window, multi-burn-rate SLO alerts.
4. Incident Management — detect → triage → mitigate → resolve → communicate → postmortem.
5. Postmortem — blameless, timeline-based, action items with owners and deadlines.
6. Toil Reduction — identify repetitive manual work; automate or eliminate (target <50% toil).
7. Chaos Engineering — steady state hypothesis → inject failure → observe → fix weaknesses.

Technology Stack

Best Practices

• SLOs based on user-facing behavior, not internal metrics.
• Error budget policy: when budget is exhausted, freeze features, fix reliability.
• Symptom-based alerting: page on "users can't check out", not "CPU > 80%".
• Every alert must be actionable — if no action needed, delete the alert.
• Blameless postmortems — focus on systemic improvements, not individuals.
• Production readiness checklist before any new service goes live.
• Target toil <50% — if operations dominate, the system needs engineering work.
• Runbooks for every alert with step-by-step mitigation guide.
• On-call load: max 2 pages/shift; more = alert hygiene problem.

Anti-Patterns to Avoid

• Alert fatigue: too many non-actionable alerts → pages get ignored.
• SLOs that nobody checks or enforces.
• Postmortems without action items, or action items without owners.
• Heroics culture: relying on specific people instead of system improvements.
• No error budget policy: reliability never gates feature work.
• Manual scaling instead of autoscaling with proper metrics.

References

• references/slo-design-guide.md — SLO definition and measurement
• references/incident-response-playbook.md — incident response process
• references/postmortem-template.md — blameless postmortem template

Expected Output Format

1. SLO definition (SLI, target, measurement window, error budget policy)
2. Alert rules (PromQL/Datadog query + runbook link)
3. Postmortem document (timeline, root cause, action items)
4. Dashboard link or configuration