gabrielmoreira SRE Practices
Apply Site Reliability Engineering principles.
Service Level Concepts
| Term | Definition | Example |
|---|---|---|
| SLI | Measured metric | Request latency p95 < 500ms |
| SLO | Target threshold for SLI | 99.9% of requests < 500ms |
| SLA | Contractual commitment (usually looser than SLO) | 99.5% uptime |
Choosing SLOs
- Pick metrics users actually care about
- Start with availability + latency + durability
- Tighten SLOs over time as reliability improves
- Don't over-constrain (cost/effort vs. benefit)
Error Budgets
Error Budget = 100% - SLO
Example: 99.9% SLO → 0.1% error budget = ~8.7 hours/month
How Error Budgets Work
- If error budget remaining → can deploy new features
- If error budget exhausted → freeze deployments, focus on reliability
- Error budget burn rate alerts trigger incident response
- Balance innovation velocity with system stability
Incident Response
Severity Levels
| Level | Definition | Response |
|---|---|---|
| SEV1 | System down, affecting many users | Immediate, all hands |
| SEV2 | Degraded but operational | 30min response |
| SEV3 | Minor issue, workaround exists | Next business day |
| SEV4 | Cosmetic, non-critical | Next sprint |
Incident Command System
- Incident Commander: Coordinates response
- Communications Lead: Status updates, stakeholder comms
- Operations Lead: Technical investigation
- Scribe: Timeline and action log
Postmortem Culture
- Blameless: systems failed, not people
- Focus on: detection, response, prevention
- Action items with owners and due dates
- Share postmortems org-wide
- Track action items to completion
Reliability Patterns
- Circuit breaker: stop cascading failures
- Bulkhead: isolate failure domains
- Retry with exponential backoff + jitter
- Rate limiting: protect against traffic spikes
- Graceful degradation: degrade features, not the whole system