Default output: return only the result, blockers, and required evidence. Omit preambles, process narration, repeated context, confidence scores, and follow-up offers. Use at most five bullets unless a required artifact or schema needs more.
Observability Engineering
Current Versions (Verify Before Use)
prometheus --version # Prometheus server
grafana-server -v # Grafana
jaeger --version # Jaeger
tempo -version # Grafana Tempo
Core Principles
- Metrics for symptoms, logs for causes, traces for paths. Use the right signal for the right question.
- Alert on symptoms, not causes. Alert when users are affected (error rate ↑, latency ↑), not when a CPU metric crosses a threshold.
- SLOs define reliability. Every service has error budget, SLO targets, and explicit consequences for budget exhaustion.
- Dashboards are for exploration, not alerting. If you need a dashboard to know something is wrong, your alerts are wrong.
- Observability data is production code. Instrumentation gets the same review rigor as business logic.
SLO Design Template
Service: <name>
SLI: <ratio of good events / total events>
SLO: <target percentage> (e.g., 99.9%)
Error Budget: 100% - SLO (e.g., 0.1% = 43.8 min/month)
Alerting:
- Fast burn: 2% budget in 1 hour → page immediately
- Slow burn: 5% budget in 6 hours → page during business hours
SLI types:
- Request-based:
good_requests / total_requests (availability, latency bucket)
- Window-based:
good_time_windows / total_time_windows (uptime)
Metric Instrumentation
RED Method (for services)
- Rate: Requests per second
- Errors: Error rate (4xx, 5xx as % of total)
- Duration: Request latency (p50, p95, p99)
USE Method (for resources)
- Utilization: % of resource used (CPU, memory, disk)
- Saturation: Queue length, wait time
- Errors: Hardware errors, failed allocations
The Four Golden Signals
- Latency
- Traffic
- Errors
- Saturation
Alert Design Rules
- Page only when human action is required immediately. Everything else is a ticket or dashboard note.
- Every alert has a runbook. If there's no runbook, there's no alert.
- Alert fatigue kills observability. If an alert fires and nobody does anything, delete the alert.
- Use multi-window, multi-burn-rate alerts. Single-threshold alerts are noisy.
Common Anti-Patterns
| Anti-Pattern |
Why It's Wrong |
Fix |
| "CPU > 80%" alert |
CPU usage is not a user symptom |
Alert on latency/error rate, investigate CPU |
| Alerting on every error |
Not all errors are user-facing |
Alert on error rate, not count |
| No SLOs |
No shared definition of "broken" |
Define SLIs and SLOs per service |
| Dashboards as primary detection |
Reactive, requires human watching |
Alert on symptoms, dashboard for diagnosis |
| Missing trace context |
Can't correlate logs/metrics/traces |
Use trace IDs in all signals |
| Log everything at INFO |
Expensive, noisy, hard to query |
Structured logs, sampled debug, ERROR for issues |
Validation Checklist
Official Resources
1---2name: monitoring3description: Design and operate application observability with metrics, logs, traces, and alerts. Use for SLO definition, dashboard design, on-call runbooks, and incident response.4---56Default output: return only the result, blockers, and required evidence. Omit preambles, process narration, repeated context, confidence scores, and follow-up offers. Use at most five bullets unless a required artifact or schema needs more.78# Observability Engineering910## Current Versions (Verify Before Use)1112```bash13prometheus --version # Prometheus server14grafana-server -v # Grafana15jaeger --version # Jaeger16tempo -version # Grafana Tempo17```1819## Core Principles20211. **Metrics for symptoms, logs for causes, traces for paths.** Use the right signal for the right question.222. **Alert on symptoms, not causes.** Alert when users are affected (error rate ↑, latency ↑), not when a CPU metric crosses a threshold.233. **SLOs define reliability.** Every service has error budget, SLO targets, and explicit consequences for budget exhaustion.244. **Dashboards are for exploration, not alerting.** If you need a dashboard to know something is wrong, your alerts are wrong.255. **Observability data is production code.** Instrumentation gets the same review rigor as business logic.2627## SLO Design Template2829```30Service: <name>31SLI: <ratio of good events / total events>32SLO: <target percentage> (e.g., 99.9%)33Error Budget: 100% - SLO (e.g., 0.1% = 43.8 min/month)34Alerting:35 - Fast burn: 2% budget in 1 hour → page immediately36 - Slow burn: 5% budget in 6 hours → page during business hours37```3839**SLI types:**40- Request-based: `good_requests / total_requests` (availability, latency bucket)41- Window-based: `good_time_windows / total_time_windows` (uptime)4243## Metric Instrumentation4445### RED Method (for services)46- **Rate:** Requests per second47- **Errors:** Error rate (4xx, 5xx as % of total)48- **Duration:** Request latency (p50, p95, p99)4950### USE Method (for resources)51- **Utilization:** % of resource used (CPU, memory, disk)52- **Saturation:** Queue length, wait time53- **Errors:** Hardware errors, failed allocations5455### The Four Golden Signals561. Latency572. Traffic583. Errors594. Saturation6061## Alert Design Rules6263- **Page only when human action is required immediately.** Everything else is a ticket or dashboard note.64- **Every alert has a runbook.** If there's no runbook, there's no alert.65- **Alert fatigue kills observability.** If an alert fires and nobody does anything, delete the alert.66- **Use multi-window, multi-burn-rate alerts.** Single-threshold alerts are noisy.6768## Common Anti-Patterns6970| Anti-Pattern | Why It's Wrong | Fix |71|---|---|---|72| "CPU > 80%" alert | CPU usage is not a user symptom | Alert on latency/error rate, investigate CPU |73| Alerting on every error | Not all errors are user-facing | Alert on error rate, not count |74| No SLOs | No shared definition of "broken" | Define SLIs and SLOs per service |75| Dashboards as primary detection | Reactive, requires human watching | Alert on symptoms, dashboard for diagnosis |76| Missing trace context | Can't correlate logs/metrics/traces | Use trace IDs in all signals |77| Log everything at INFO | Expensive, noisy, hard to query | Structured logs, sampled debug, ERROR for issues |7879## Validation Checklist8081- [ ] Every service exports RED metrics82- [ ] Every service has defined SLOs with error budgets83- [ ] Every page alert has a tested runbook84- [ ] Alert routing goes to the right team (not a catch-all)85- [ ] Dashboards answer "what happened" and "why" for known failure modes86- [ ] Traces span service boundaries with propagated context87- [ ] Log retention and cost are monitored8889## Official Resources9091- [Google SRE Book — SLOs](https://sre.google/sre-book/service-level-objectives/)92- [Prometheus best practices](https://prometheus.io/docs/practices/)93- [Grafana alerting docs](https://grafana.com/docs/grafana/latest/alerting/)94- [OpenTelemetry](https://opentelemetry.io/docs/)95- [Jaeger tracing](https://www.jaegertracing.io/docs/)
v1truv1us