# Sre

> Site reliability engineering practices

- Skill: `neuralblitz/sre` (Agent Skill)
- Install (CLI): `npx skillmds@latest add neuralblitz/sre`
- Raw SKILL.md: https://api.skillmd.com/api/skills/neuralblitz/sre/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: NeuralBlitz (https://skillmd.com/u/neuralblitz)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/neuralblitz/sre

---


## What I do

- Apply SRE principles to systems reliability
- Define and measure reliability metrics
- Implement monitoring and alerting
- Respond to incidents and outages
- Automate operational processes
- Conduct chaos engineering experiments

## When to use me

- When building reliable distributed systems
- When establishing reliability practices
- When managing production systems
- When responding to incidents
- When automating operations
- When measuring system health

## Key Concepts

### Error Budgets

```yaml
# Define error budget
slo:
  service: payment-api
  sli: availability
  target: 99.9%
  window: 30 days
  
# Error budget calculation
# 30 days * 24 hours * 60 minutes = 43,200 minutes
# 0.1% error budget = 43.2 minutes of allowed downtime

# Error budget alerts
- alert: ErrorBudgetWarning
  expr: |
    (
      1 - (
        sum(rate(http_requests_total{service="payment-api"}[30d])) 
        - sum(rate(http_requests_total{service="payment-api",status=~"5.."}[30d]))
      ) / sum(rate(http_requests_total{service="payment-api"}[30d]))
    ) > 0.05
  for: 1h
```

### Incident Management

```python
# Incident response workflow
class IncidentManager:
    def __init__(self):
        self.escalation_levels = {
            0: ['team-slack'],
            1: ['team-slack', 'on-call'],
            2: ['team-slack', 'on-call', 'manager'],
            3: ['team-slack', 'on-call', 'manager', 'executive']
        }
        
    def create_incident(self, severity: int):
        incident = Incident(
            severity=severity,
            status='investigating',
            timeline=[TimelineEvent('Created', now())]
        )
        
        self.notify(incident, self.escalation_levels[severity])
        return incident
        
    def update_status(self, incident, status: str):
        incident.status = status
        incident.timeline.append(TimelineEvent(status, now()))
        
        if status == 'resolved':
            incident.duration = now() - incident.created_at
            self.schedule_postmortem(incident)
```

### Toil Analysis

```yaml
# Toil classification
toil_categories:
  manual_deployments:
    frequency: 10/week
    duration_minutes: 15
    toil_hours: 2.5/week
    
  data_cleanup:
    frequency: 5/week
    duration_minutes: 30
    toil_hours: 2.5/week
    
  certificate_renewal:
    frequency: 1/month
    duration_minutes: 60
    toil_hours: 1/month
    
# Total toil: ~5 hours/week
# Goal: Reduce to <2 hours/week via automation
```

### Service Health

```python
# Health check implementation
class ServiceHealth:
    def __init__(self):
        self.checks = []
        
    def register_check(self, name: str, check_fn):
        self.checks.append({'name': name, 'check': check_fn})
        
    async def get_health(self):
        results = []
        overall_healthy = True
        
        for check in self.checks:
            try:
                result = await check['check']()
                results.append({
                    'name': check['name'],
                    'healthy': result.healthy,
                    'details': result.details
                })
                if not result.healthy:
                    overall_healthy = False
            except Exception as e:
                results.append({
                    'name': check['name'],
                    'healthy': False,
                    'error': str(e)
                })
                overall_healthy = False
                
        return HealthStatus(healthy=overall_healthy, checks=results)
```

### Reliability Engineering

- **Blameless Postmortems**: Focus on system improvement
- **Proactive Reliability**: Build reliability in, don't add later
- **SLOs over SLAs**: Internal goals stricter than customer promises
- **Error Budgets**: Balance reliability with velocity
- **Reduce MTTR**: Focus on detection and response time
- **Capacity Planning**: Plan for growth

### Runbook Example

```yaml
# runbook: high-cpu-alert
name: High CPU Usage
severity: warning

description: |
  One or more instances have high CPU usage.

steps:
  - name: Identify affected instances
    command: |
      kubectl top nodes
      
  - name: Check for traffic spike
    command: |
      kubectl get pods --sort-by='.spec.containers[0].resources.limits.cpu'
      
  - name: Check for runaway processes
    command: |
      kubectl exec -it <pod> -- top
      
  - name: Scale horizontally if needed
    command: |
      kubectl scale deployment <name> --replicas=5
      
  - name: Check HPA if enabled
    command: |
      kubectl get hpa
```

### Key SRE Practices

1. **Define SLIs**: What matters to users
2. **Set SLOs**: Target reliability levels
3. **Monitor SLIs**: Measure continuously
4. **Alert on SLO Burn**: Proactive warnings
5. **Review Incidents**: Blameless postmortems
6. **Automate**: Reduce toil continuously