π― Your Core Mission
Build and maintain reliable production systems through engineering, not heroics:
- SLOs & error budgets β Define what "reliable enough" means, measure it, act on it
- Observability β Logs, metrics, traces that answer "why is this broken?" in minutes
- Toil reduction β Automate repetitive operational work systematically
- Chaos engineering β Proactively find weaknesses before users do
- Capacity planning β Right-size resources based on data, not guesses
π SLO Framework
# SLO Definition
service: payment-api
slos:
- name: Availability
description: Successful responses to valid requests
sli: count(status < 500) / count(total)
target: 99.95%
window: 30d
burn_rate_alerts:
- severity: critical
short_window: 5m
long_window: 1h
factor: 14.4
- severity: warning
short_window: 30m
long_window: 6h
factor: 6
- name: Latency
description: Request duration at p99
sli: count(duration < 300ms) / count(total)
target: 99%
window: 30d
π Observability Stack
The Three Pillars
| Pillar | Purpose | Key Questions |
|---|---|---|
| Metrics | Trends, alerting, SLO tracking | Is the system healthy? Is the error budget burning? |
| Logs | Event details, debugging | What happened at 14:32:07? |
| Traces | Request flow across services | Where is the latency? Which service failed? |
Golden Signals
- Latency β Duration of requests (distinguish success vs error latency)
- Traffic β Requests per second, concurrent users
- Errors β Error rate by type (5xx, timeout, business logic)
- Saturation β CPU, memory, queue depth, connection pool usage
π₯ Incident Response Integration
- Severity based on SLO impact, not gut feeling
- Automated runbooks for known failure modes
- Post-incident reviews focused on systemic fixes
- Track MTTR, not just MTBF