aAAaqwq Google SRE Practices
Source: Google SRE Book + SRE Workbook + The Site Reliability Workbook
- Practical Google/ByteDance production operations experience Core Philosophy: Reliability is a feature, and like any feature, it has a cost. SRE makes the trade-off explicit via Error Budgets.
Core Principles
┌─────────────────────────────────────────────────────────────────┐
│ │
│ Google SRE's core insight: │
│ ◆ 100% reliability is the WRONG target (impossible cost) │
│ ◆ Error Budgets make reliability a PRODUCT decision, not free │
│ ◆ Automation > Manual (toil is the enemy of SRE) │
│ ◆ Everything fails — design for failure, not for perfection │
│ ◆ Observability requires three pillars: logs, metrics, traces │
│ │
└─────────────────────────────────────────────────────────────────┘
1. SLI / SLO / SLA — The Service Level Triad
1.1 Definitions
SLI (Service Level Indicator):
A quantifiable measure of some aspect of service level.
Examples: request latency, error rate, throughput, availability.
GOOD SLI: "fraction of requests completed in < 200ms"
BAD SLI: "user happiness" (can't measure)
BAD SLI: "average latency" (hides p99 problems under p50)
SLO (Service Level Objective):
A target value or range for a service level.
Examples: 99.9% availability, p99 latency < 200ms.
⚠️ Design for reality, not hubris:
❌ "We'll aim for 99.99% because Google does it!"
✅ "The business needs 99.9% and we can sustain that"
SLA (Service Level Agreement):
A promise to your customers about the level of service.
Usually with consequences (credits, penalties).
Always stricter than SLOs (SLA < SLO, because you need a buffer).
1.2 How to Define SLIs
Start by asking: "What matters to our users?"
For a web API service:
Availability: % of HTTP requests that return 2xx/3xx
Latency: % of requests completing in < 200ms (p99)
Throughput: Requests per second sustained
Durability: % of writes not lost (for storage)
For a data pipeline:
Freshness: Age of most recent data
Correctness: % of records passing validation
Completeness: % of expected data actually delivered
Latency: Time from event → available for query
For an ML inference service:
Availability: % of inference requests served successfully
Latency: p99 inference time
Accuracy: Model quality metric (e.g., BLEU, perplexity)
Freshness: Age of model since last training
1.3 Choosing SLO Targets — The Hard Part
SLO Selection Process:
1. Set an initial target (e.g., 99.9%)
2. Ask: "If customer satisfaction dropped 1%, would this target cause it?"
If YES → Raise the target
If NO → Lower the target
3. Ask: "Can we sustain this target for 3 months without burning out?"
If NO → Lower the target (unsustainable = undeliverable)
If YES → Accept the target
4. Monitor for 1 quarter, then adjust
→ If we never miss SLO → it's too loose
→ If we always miss SLO → it's too tight
→ Goldilocks zone: hit SLO 95-99% of the time
Magic formula: Good SLO = targets that keep customers happy
without creating false urgency.
1.4 Common SLO Patterns
┌──────────────────────────────────────────────────────────────┐
│ │ Tight (< 99.9%) │ SLO Target │
├──────────────────────────┼─────────────────┼─────────────────┤
│ Internal/Cost-sensitive │ 99% │ 99.5% │
│ Customer-facing │ 99.9% │ 99.99% │
│ Critical infra │ 99.99% │ 99.999% │
│ Money-related │ 99.999% │ 99.9999% │
└──────────────────────────┴─────────────────┴─────────────────┘
9s translate to downtime per year:
99% → 3.65 days/year
99.9% → 8.76 hours/year
99.99% → 52.6 minutes/year
99.999% → 5.26 minutes/year
99.9999% → 31.5 seconds/year
2. Error Budgets — The SRE Superpower
2.1 What is an Error Budget?
Error Budget = 1 − SLO
If your SLO is 99.9% availability:
Error Budget = 0.1% of total requests = your "allowed failure"
Think of it as:
"You have X failures budgeted for this month.
Spend them wisely."
After the budget is spent:
→ Release velocity slows down (stability mode)
→ Before budget spent:
→ Full speed ahead (feature mode)
2.2 How Error Budgets Transform Decision-Making
Without Error Budget:
❌ "Can we deploy? I don't know, let's see if it feels risky"
❌ Ops says "no" → feature team feels blocked
❌ No explicit trade-off — everything is crisis
With Error Budget:
✅ "We have 10% budget left this month — safe to deploy"
✅ "We have 0% budget left — NO DEPLOYS until next month"
✅ Trade-off is explicit, data-driven, not political
✅ Product owns the budget: "Want more releases? Accept more risk"
2.3 Error Budget Policy Template
┌──────────────────────────────────────────────────────────────┐
│ Error Budget Policy │
│ │
│ Budget Period: Calendar month (30 day rolling window) │
│ │
│ Budget burned thresholds: │
│ ┌────────────┬──────────┬────────────────────────────────┐ │
│ │ Threshold │ Action │ Owner │ │
│ ├────────────┼──────────┼────────────────────────────────┤ │
│ │ < 50% │ Normal │ Team makes own decisions │ │
│ │ 50-80% │ Caution │ Feature freeze unless critical│ │
│ │ 80-100% │ Alert │ Postmortem required │ │
│ │ > 100% │ Freeze │ NO deploys. Reliability mode │ │
│ └────────────┴──────────┴────────────────────────────────┘ │
│ │
│ Budget reset: Beginning of each calendar month │
└──────────────────────────────────────────────────────────────┘
3. Monitoring & Alerting — The Golden Signals
3.1 Google's Four Golden Signals
┌──────────────────────────────────────────────────────────────┐
│ ① Latency Time to service a request │
│ → Distinguish "fast errors" from "slow │
│ success" (both are bad, different fix)│
│ → Track p50, p95, p99 │
│ │
│ ② Traffic Demand on your system │
│ → Requests/sec, connections, QPS │
│ → Track per route/endpoint │
│ │
│ ③ Errors Rate of failed requests │
│ → Explicit 5xx, implicit (200 but wrong)│
│ → Track per error code + per endpoint │
│ │
│ ④ Saturation How "full" your service is │
│ → CPU, memory, disk, network I/O │
│ → Connection pool, queue depth │
│ → Best leading indicator of trouble │
└──────────────────────────────────────────────────────────────┘
3.2 Alert Severity Classification
┌─────────┬──────────────────────────────────┬──────────────────┐
│ Level │ When to use it │ Response │
├─────────┼──────────────────────────────────┼──────────────────┤
│ P0/Page │ User-facing outage │ Wake the person │
│ │ SLO breach likely in < 10 min │ │
│ │ Data loss risk │ │
├─────────┼──────────────────────────────────┼──────────────────┤
│ P1/Ticket│ Error budget > 50% burned │ Next business day │
│ │ Non-critical degradation │ │
│ │ Capacity threshold warning │ │
├─────────┼──────────────────────────────────┼──────────────────┤
│ P2/Log │ Non-urgent anomaly │ Next sprint │
│ │ Performance drift │ │
│ │ Deprecation warnings │ │
└─────────┴──────────────────────────────────┴──────────────────┘
3.3 Alert Design Principles
Rule 1: Every alert must be actionable.
❌ "CPU at 80%" — what do you want me to DO?
✅ "CPU at 80% with traffic rising" — scale up or investigate.
Rule 2: Alert on symptoms (user-facing), not causes (internal).
❌ "Redis memory high" — I don't care about Redis
✅ "Request timeout rate > 1%" — NOW I care
Rule 3: No alert should be "noise more than twice."
If an alert fires and the response is "ignore it" 2x in a row,
either fix the alert or delete it. Alert fatigue kills.
Rule 4: Every alert must have a runbook.
The runbook doesn't need to be 10 pages, but it must answer:
- "What's broken?" (symptom)
- "How do I confirm it?" (diagnostic step)
- "What do I do?" (mitigation)
- "When do I escalate?" (threshold)
3.4 Runbook Template
┌──────────────────────────────────────────────────────────────┐
│ Alert: High Error Rate (>1% 5xx) │
│ │
│ 1. CHECK │
│ `kubectl get pods` → any pods restarting? │
│ `curl -I /health` → service alive? │
│ `tail -100 logs/app.log` → any error stack? │
│ │
│ 2. TRIAGE │
│ → If all pods healthy → check upstream dependencies │
│ → If pods crashing → `kubectl describe pod` for reason │
│ → If upstream failing → escalate to dependency team │
│ │
│ 3. MITIGATE │
│ → If dependency issue: circuit breaker / fallback │
│ → If code issue: rollback to last known good image │
│ → If data issue: database rollback / recovery │
│ │
│ 4. ESCALATE │
│ → If not resolved in 15 min → page senior SRE │
│ → If data loss suspected → page DBA + engineering lead │
│ → If security breach → security team + legal │
└──────────────────────────────────────────────────────────────┘
4. Incident Response — Triage & Manage
4.1 Incident Command System
┌──────────────────────────────────────────────────────────────┐
│ INCIDENT COMMAND STRUCTURE │
│ │
│ IC (Incident Commander): │
│ × NOT responsible for fixing │
│ ✓ Coordinates response, communications, timeline │
│ ✓ Decides when to call in more people │
│ ✓ Decides when to declare incident over │
│ │
│ Ops Lead: │
│ ✓ Directs technical mitigation │
│ ✓ Coordinates with IC on progress │
│ │
│ Scribe: │
│ ✓ Documents timeline and actions │
│ ✓ Relieves IC from note-taking │
│ │
│ Comms Lead: │
│ ✓ Internal/external communications │
│ ✓ Status updates to stakeholders │
└──────────────────────────────────────────────────────────────┘
4.2 Incident Triage Process
1. STOP THE BLEEDING (first 5 min)
→ Is there automatic mitigation? (circuit breaker, scale-up)
→ If not: rollback, restart, or redirect traffic
→ Don't investigate root cause yet — stabilize first
2. STABILIZE (5-30 min)
→ Implement immediate workaround
→ Restore service to acceptable state
→ Document what was done
3. INVESTIGATE (30 min +)
→ Root cause analysis begins
→ Use timeline to track discoveries
→ Avoid blame — focus on systems
4. RESOLVE
→ Permanent fix deployed
→ Monitoring confirms normal operation
→ Incident declared over
5. POSTMORTEM (within 1 week)
→ Full timeline
→ 5 Whys
→ Action items with owners + due dates
→ Blameless review
5. Chaos Engineering — Proactive Reliability
5.1 Core Principle
"Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system's capability to withstand turbulent conditions in production." — Netflix
5.2 The Scientific Method for Production
Step 1: Define "steady state" → measurable output (e.g., p99 < 200ms)
Step 2: Hypothesize → "If instance fails, system degrades by at most 5%"
Step 3: Introduce failure → Kill a pod, inject latency, blackhole traffic
Step 4: Measure → Compare actual behavior against steady state
Step 5: Fix or Confirm →
If hypothesis correct → system is resilient
If incorrect → you found a weakness — fix it
5.3 Practical Chaos Experiments
Level 1: Infrastructure Failure
✅ Kill a randomly selected pod
✅ Network partition a service
✅ Kill a database replica
✅ Simulate AZ (availability zone) outage
Level 2: Service Degradation
✅ Inject latency on inter-service calls
✅ Simulate DNS failure
✅ Rate-limit a dependency
✅ Certificate expiration simulation
Level 3: Data Plane Issues
✅ Corrupt a cache entry
✅ Simulate authentication service timeout
✅ Media/image processing service slowdown
✅ Dead-letter queue overflow
Level 4: Compound Failures
✅ Two simultaneous AZ outages
✅ Cascading failure: DB slow → cache miss → DB even slower
✅ Config pushed to wrong region
5.4 Chaos Engineering Safety
Golden Rules:
1. NEVER experiment during business peak hours
2. Start with small blast radius (1 pod, not 50)
3. Have automatic rollback:
→ If error rate > threshold → auto-stop experiment
4. Communicate before running experiments
5. Every experiment has explicit hypothesis + expected outcome
6. Blast radius must be limited to non-critical path initially
6. Capacity Planning
6.1 The Capacity Planning Process
┌──────────────────────────────────────────────────────────────┐
│ Capacity Planning Cycle (Quarterly) │
│ │
│ Month 1: Demand Forecasting │
│ → Analyze historical growth curves │
│ → Business input: new features, campaigns, events │
│ → Traffic projection: "How much load in 6 months?" │
│ │
│ Month 2: Resource Modeling │
│ → Model capacity needs per resource type │
│ → CPU, memory, disk, network, database connections │
│ → Size per service, per environment │
│ │
│ Month 3: Procurement / Scaling │
│ → Cloud resources: auto-scaling config updates │
│ → Budget approval for additional resources │
│ → Infrastructure-as-code changes to support growth │
└──────────────────────────────────────────────────────────────┘
6.2 Back-of-Envelope Capacity Estimation
Given: 10M MAU, 100M requests/day, 500ms avg request time
Peak traffic (assuming 2x daily peak factor):
100M / 86400 sec ≈ 1157 req/sec average
1157 × 2 = ~2300 req/sec peak
Compute needed:
2300 req/sec × 0.5 sec = 1150 concurrent requests
Each request uses ~200MB memory
Need: 1150 × 200MB ≈ 230GB RAM total
With 8GB per instance: 29 instances ≈ 30
With 2x headroom: 60 instances
Cost estimate:
60 × $0.10/hr × 24 × 30 = $4320/month (compute only)
6.3 Autoscaling Strategy
┌─────────────┬──────────────────────────────────────────────────┐
│ Metric │ HPA Configuration │
├─────────────┼──────────────────────────────────────────────────┤
│ CPU │ Scale at 70% utilization │
│ Memory │ Scale at 80% utilization │
│ Req/sec │ Scale at 1000 req/sec per pod │
│ Queue depth │ Scale at 100 messages per worker per second │
│ Custom │ Scale on SLO burn rate (error budget consumed) │
└─────────────┴──────────────────────────────────────────────────┘
⚠️ Beware of thrashing: add cooldown periods (3-5 min)
⚠️ Scale up fast (30s), scale down slow (5min)
⚠️ Have min/max instance limits (never scale to 0 for critical)
7. Toil — The Silent Killer
7.1 What is Toil?
Toil = operational work that is:
✓ Manual (requires human touch)
✓ Repetitive (same thing over and over)
✓ Automatable (if time were invested)
✓ Tactical (reactive, not strategic)
✓ No enduring value (the world is the same after you do it)
Examples:
❌ Restarting crashed services manually
❌ Copy-paste deploys
❌ Manually generating reports
❌ SSH-ing into servers to check logs
❌ Manually rotating credentials
NOT toil:
✓ Writing automation
✓ Debugging novel issues
✓ Architecture design
✓ Code review
✓ On-call rotation (necessary, not toil)
7.2 Toil Budget
Google SRE goal: < 50% of work hours on toil
Aspirational goal: < 25% (leaves 50% for engineering)
Track toil per team:
┌──────────────────────────────────────────────────────────────┐
│ Week ending | Total hours | Toil hours | Toil % | Actions │
├──────────────┼─────────────┼────────────┼────────┼──────────┤
│ May 1 │ 160 │ 72 │ 45% │ Auto-deploy│
│ May 8 │ 160 │ 60 │ 37% │ CI pipeline│
│ May 15 │ 160 │ 48 │ 30% │ Done │
└──────────────────────────────────────────────────────────────┘
7.3 Automation Priority Framework
When deciding what to automate, use this matrix:
┌──────────────────────────────────────────────────────────────┐
│ Frequency │
│ High │ Low │
├───────────────────────────────┼──────────────────────────────┤
│H │ AUTOMATE NOW! │ INVESTIGATE │
│I │ Monthly report generation │ Bi-annual cert rotation │
│G │ Deployment │ Quarterly snapshot restore │
│H │ Incident triage │ │
├───────────────────────────────┼──────────────────────────────┤
│L │ QUICK WIN: Document │ IGNORE │
│O │ Once-a-week password reset│ Annual manual data cleanup │
│W │ Common ticket responses │ Yearly hardware audit │
└───────────────────────────────┴──────────────────────────────┘
8. Reliability Maturity Model
Level 0: Hero Ops
├── No SLOs, no monitoring
├── "We fix it when it breaks"
└── Page every time, chaos all the time
Level 1: Reactive
├── Basic monitoring (CPU, memory, disk)
├── Runbooks for common incidents
├── Manual deploys
└── MTTR: hours to days
Level 2: Proactive
├── SLOs defined for critical services
├── Error budgets tracked
├── CI/CD with automated testing
├── Incident command structure
└── MTTR: < 1 hour
Level 3: Automated
├── Autoscaling for capacity
├── Auto-remediation for common failures
├── Chaos engineering experiments
├── Capacity planning automated
└── MTTR: < 15 minutes
Level 4: Self-Healing
├── System adapts to failures automatically
├── Proactive scaling based on leading indicators
├── Automated postmortem analysis
├── Self-service reliability tools
└── MTTR: < 5 minutes
Where We Are vs Google
┌──────────────┬───────────────┬────────────────┐
│ Capability │ Our Level │ Google Target │
├──────────────┼───────────────┼────────────────┤
│ Monitoring │ L1-L2 │ L3 │
│ Alerting │ L1 │ L3 │
│ SLO/Error Budget│ L0 Not yet │ L3 │
│ Incident Mgmt │ L1 │ L3-L4 │
│ Chaos Eng │ L0 │ L3 │
│ Automation │ L1 │ L3-L4 │
│ Capacity Plan│ L0 │ L2-L3 │
│ Toil Mgmt │ L0 │ L2 │
└──────────────┴───────────────┴────────────────┘
References
references/incident-runbooks.md— Pre-built runbook templates for common incidentsreferences/alert-rules-examples.md— Practical Prometheus/Grafana alert rule examplesreferences/chaos-experiment-templates.md— Chaos experiment design templates with blast radius controlsreferences/capacity-planning-examples.md— Back-of-envelope examples for different service patterns