Default output: return only the result, blockers, and required evidence. Omit preambles, process narration, repeated context, confidence scores, and follow-up offers. Use at most five bullets unless a required artifact or schema needs more.
Observability Engineering
Current Versions (Verify Before Use)
prometheus --version # Prometheus server
grafana-server -v # Grafana
jaeger --version # Jaeger
tempo -version # Grafana Tempo
Core Principles
- Metrics for symptoms, logs for causes, traces for paths. Use the right signal for the right question.
- Alert on symptoms, not causes. Alert when users are affected (error rate ↑, latency ↑), not when a CPU metric crosses a threshold.
- SLOs define reliability. Every service has error budget, SLO targets, and explicit consequences for budget exhaustion.
- Dashboards are for exploration, not alerting. If you need a dashboard to know something is wrong, your alerts are wrong.
- Observability data is production code. Instrumentation gets the same review rigor as business logic.
SLO Design Template
Service: <name>
SLI: <ratio of good events / total events>
SLO: <target percentage> (e.g., 99.9%)
Error Budget: 100% - SLO (e.g., 0.1% = 43.8 min/month)
Alerting:
- Fast burn: 2% budget in 1 hour → page immediately
- Slow burn: 5% budget in 6 hours → page during business hours
SLI types:
- Request-based:
good_requests / total_requests (availability, latency bucket)
- Window-based:
good_time_windows / total_time_windows (uptime)
Metric Instrumentation
RED Method (for services)
- Rate: Requests per second
- Errors: Error rate (4xx, 5xx as % of total)
- Duration: Request latency (p50, p95, p99)
USE Method (for resources)
- Utilization: % of resource used (CPU, memory, disk)
- Saturation: Queue length, wait time
- Errors: Hardware errors, failed allocations
The Four Golden Signals
- Latency
- Traffic
- Errors
- Saturation
Alert Design Rules
- Page only when human action is required immediately. Everything else is a ticket or dashboard note.
- Every alert has a runbook. If there's no runbook, there's no alert.
- Alert fatigue kills observability. If an alert fires and nobody does anything, delete the alert.
- Use multi-window, multi-burn-rate alerts. Single-threshold alerts are noisy.
Common Anti-Patterns
| Anti-Pattern |
Why It's Wrong |
Fix |
| "CPU > 80%" alert |
CPU usage is not a user symptom |
Alert on latency/error rate, investigate CPU |
| Alerting on every error |
Not all errors are user-facing |
Alert on error rate, not count |
| No SLOs |
No shared definition of "broken" |
Define SLIs and SLOs per service |
| Dashboards as primary detection |
Reactive, requires human watching |
Alert on symptoms, dashboard for diagnosis |
| Missing trace context |
Can't correlate logs/metrics/traces |
Use trace IDs in all signals |
| Log everything at INFO |
Expensive, noisy, hard to query |
Structured logs, sampled debug, ERROR for issues |
Validation Checklist
Official Resources
1---2name: monitoring3description: Design and operate application observability with metrics, logs, traces, and alerts. Use for SLO definition, dashboard design, on-call runbooks, and incident response.4metadata:5 category: user-invoked6disable-model-invocation: true7---89Default output: return only the result, blockers, and required evidence. Omit preambles, process narration, repeated context, confidence scores, and follow-up offers. Use at most five bullets unless a required artifact or schema needs more.1011# Observability Engineering1213## Current Versions (Verify Before Use)1415```bash16prometheus --version # Prometheus server17grafana-server -v # Grafana18jaeger --version # Jaeger19tempo -version # Grafana Tempo20```2122## Core Principles23241. **Metrics for symptoms, logs for causes, traces for paths.** Use the right signal for the right question.252. **Alert on symptoms, not causes.** Alert when users are affected (error rate ↑, latency ↑), not when a CPU metric crosses a threshold.263. **SLOs define reliability.** Every service has error budget, SLO targets, and explicit consequences for budget exhaustion.274. **Dashboards are for exploration, not alerting.** If you need a dashboard to know something is wrong, your alerts are wrong.285. **Observability data is production code.** Instrumentation gets the same review rigor as business logic.2930## SLO Design Template3132```33Service: <name>34SLI: <ratio of good events / total events>35SLO: <target percentage> (e.g., 99.9%)36Error Budget: 100% - SLO (e.g., 0.1% = 43.8 min/month)37Alerting:38 - Fast burn: 2% budget in 1 hour → page immediately39 - Slow burn: 5% budget in 6 hours → page during business hours40```4142**SLI types:**43- Request-based: `good_requests / total_requests` (availability, latency bucket)44- Window-based: `good_time_windows / total_time_windows` (uptime)4546## Metric Instrumentation4748### RED Method (for services)49- **Rate:** Requests per second50- **Errors:** Error rate (4xx, 5xx as % of total)51- **Duration:** Request latency (p50, p95, p99)5253### USE Method (for resources)54- **Utilization:** % of resource used (CPU, memory, disk)55- **Saturation:** Queue length, wait time56- **Errors:** Hardware errors, failed allocations5758### The Four Golden Signals591. Latency602. Traffic613. Errors624. Saturation6364## Alert Design Rules6566- **Page only when human action is required immediately.** Everything else is a ticket or dashboard note.67- **Every alert has a runbook.** If there's no runbook, there's no alert.68- **Alert fatigue kills observability.** If an alert fires and nobody does anything, delete the alert.69- **Use multi-window, multi-burn-rate alerts.** Single-threshold alerts are noisy.7071## Common Anti-Patterns7273| Anti-Pattern | Why It's Wrong | Fix |74|---|---|---|75| "CPU > 80%" alert | CPU usage is not a user symptom | Alert on latency/error rate, investigate CPU |76| Alerting on every error | Not all errors are user-facing | Alert on error rate, not count |77| No SLOs | No shared definition of "broken" | Define SLIs and SLOs per service |78| Dashboards as primary detection | Reactive, requires human watching | Alert on symptoms, dashboard for diagnosis |79| Missing trace context | Can't correlate logs/metrics/traces | Use trace IDs in all signals |80| Log everything at INFO | Expensive, noisy, hard to query | Structured logs, sampled debug, ERROR for issues |8182## Validation Checklist8384- [ ] Every service exports RED metrics85- [ ] Every service has defined SLOs with error budgets86- [ ] Every page alert has a tested runbook87- [ ] Alert routing goes to the right team (not a catch-all)88- [ ] Dashboards answer "what happened" and "why" for known failure modes89- [ ] Traces span service boundaries with propagated context90- [ ] Log retention and cost are monitored9192## Official Resources9394- [Google SRE Book — SLOs](https://sre.google/sre-book/service-level-objectives/)95- [Prometheus best practices](https://prometheus.io/docs/practices/)96- [Grafana alerting docs](https://grafana.com/docs/grafana/latest/alerting/)97- [OpenTelemetry](https://opentelemetry.io/docs/)98- [Jaeger tracing](https://www.jaegertracing.io/docs/)
v1truv1us