Site Reliability

What I do

• Implement SRE practices and principles
• Define and track SLIs and SLOs
• Design error budgets and alerting
• Conduct post-incident reviews
• Automate operational tasks
• Build observability into systems

When to use me

• When implementing reliability practices
• When defining service level objectives
• When managing incident response
• When automating operations
• When improving system reliability
• When building SLO dashboards

Key Concepts

SLI/SLO/SLA Framework

# Service Level Indicators
slis:
  - name: request-availability
    description: "Ratio of successful requests to total requests"
    type: ratio
    source: prometheus
    query: |
      sum(rate(http_requests_total{status!~"5.."}[5m])) 
      / sum(rate(http_requests_total[5m]))

  - name: request-latency
    description: "P99 latency in milliseconds"
    type: threshold
    source: prometheus
    query: |
      histogram_quantile(0.99, 
        sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
      ) * 1000

# Service Level Objectives
slo:
  - name: api-availability
    sli: request-availability
    target: 99.9
    window: 30d
    
  - name: api-latency
    sli: request-latency
    target: 500
    window: 7d

# Service Level Agreement
sla:
  - name: api-availability
    objective: api-availability
    target: 99.9
    link: https://example.com/sla

Error Budget Policy

# Alert on error budget burn rate
- alert: ErrorBudgetBurnRateHigh
  expr: |
    sum(rate(http_requests_total{slo="api-availability"}[5m])) > 14.4
    / 1000
  for: 1h
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning too fast"
    description: "Error budget burn rate > 10x for 1 hour"

# Alert on remaining error budget
- alert: ErrorBudgetExhausted
  expr: |
    (slo:api-availability:ratio{job="api"}/ 1) < 0.9
  for: 1h
  labels:
    severity: critical
  annotations:
    summary: "Error budget exhausted"
    description: "Less than 10% of error budget remaining"

Toil Reduction

# Automating manual work
class AutomatedOps:
    def __init__(self):
        self.alert_handlers = {
            'high_cpu': self.handle_high_cpu,
            'disk_full': self.handle_disk_full,
            'error_rate': self.handle_error_rate
        }
        
    def handle_alert(self, alert):
        handler = self.alert_handlers.get(alert.type)
        if handler:
            # Attempt automated remediation
            result = handler(alert)
            if result.success:
                return {'action': 'auto_resolved'}
            else:
                # Escalate if automated fix failed
                return {'action': 'escalate', 'reason': result.error}
        return {'action': 'no_handler'}
        
    def handle_high_cpu(self, alert):
        # Check if auto-scaling can help
        if self.can_scale(alert.target):
            self.scale_out(alert.target)
            return Result(success=True)
        return Result(success=False, error='Cannot scale')

Post-Incident Review

# Incident Review: Service Outage

## Summary
- **Date**: 2024-01-15
- **Duration**: 45 minutes
- **Impact**: 15% of users unable to access API

## Timeline
- 10:00 - Alert triggered (high error rate)
- 10:05 - On-call paged
- 10:10 - Investigation started
- 10:25 - Root cause identified (database connection pool exhaustion)
- 10:35 - Fix deployed
- 10:45 - Service recovered

## Root Cause
Database connection pool was undersized for traffic spike.

## Action Items
- [ ] Increase connection pool size
- [ ] Add circuit breaker pattern
- [ ] Create load test for connection handling
- [ ] Update runbook for database scaling

## Lessons Learned
1. Need better observability into DB connection pool
2. Alert threshold was too high
3. Manual scaling takes too long

Observability for SRE

# Custom SRE metrics
class SRERecorder:
    def __init__(self, prometheus_client):
        self.errors = Counter('sre_errors_total', 'Total errors')
        self.latency = Histogram('sre_latency_seconds', 'Request latency')
        self.availability = Gauge('sre_availability_ratio', 'Current availability')
        
    def record_request(self, success: bool, duration: float):
        if not success:
            self.errors.inc()
        self.latency.observe(duration)
        
    def calculate_availability(self, window_minutes: int):
        # Calculate rolling availability
        pass

SRE Metrics (DORA)

Core SRE Principles

• Embrace Risk: Not 100% reliability required
• SLOs Drive Work: Prioritize based on objectives
• Eliminate Toil: Automate repetitive work
• Measure Everything: Data-driven decisions
• Reduce Waste: Don't over-engineer
• Shared Ownership: One team runs what they build