1---2name: observability-design3description: Design observability systems — metrics/logs/traces strategy, instrumentation standards, correlation IDs, sampling, dashboards, and alert design. TRIGGER when: user says /observability-design, needs to improve system observability, or asks about monitoring, tracing, or logging strategy.4---56# Observability Design78You are an SRE specialist. Design an observability strategy that provides actionable insight into system behavior across metrics, logs, and traces.910## Process1112### Step 1: Define Observability Goals13| Goal | Metric |14|------|--------|15| Mean time to detect (MTTD) | < X minutes |16| Mean time to diagnose (MTTD) | < X minutes |17| Alert accuracy | > 90% actionable (low false positives) |18| Coverage | 100% of critical paths instrumented |1920### Step 2: Design the Three Pillars21**Metrics:**22| Category | Examples | Tool |23|----------|---------|------|24| Infrastructure | CPU, memory, disk, network | Prometheus, CloudWatch |25| Application | Request rate, error rate, duration (RED) | Prometheus, StatsD |26| Business | Orders/min, signups, revenue | Custom metrics |2728**Logs:**29| Standard | Details |30|----------|---------|31| Format | Structured JSON with consistent fields |32| Fields | timestamp, level, service, trace_id, message, context |33| Levels | DEBUG, INFO, WARN, ERROR, FATAL |34| Retention | Hot: 7 days, Warm: 30 days, Cold: 90 days |3536**Traces:**37| Element | Implementation |38|---------|---------------|39| Propagation | W3C TraceContext headers across all services |40| Sampling | 100% for errors, 10% for normal traffic |41| Span naming | `service.operation` (e.g., `api.getUser`) |42| Attributes | user_id, request_id, feature_flags |4344### Step 3: Instrument Services45| Layer | What to Instrument |46|-------|-------------------|47| HTTP/gRPC | Request count, latency histogram, error rate |48| Database | Query duration, connection pool, slow queries |49| Cache | Hit/miss ratio, latency, eviction rate |50| Queue | Depth, processing time, dead letter count |51| External APIs | Latency, error rate, circuit breaker state |5253### Step 4: Design Dashboards54| Dashboard | Audience | Content |55|-----------|----------|---------|56| Service overview | On-call SRE | RED metrics, error budget, SLO status |57| Infrastructure | Platform team | Resource utilization, capacity |58| Business | Product/exec | Business KPIs, feature adoption |59| Debug | Engineers | Detailed per-service metrics, trace links |6061### Step 5: Configure Alerting62| Severity | Criteria | Response |63|----------|----------|----------|64| P1 | SLO burn rate > 10× | Page on-call immediately |65| P2 | SLO burn rate > 2× | Page during business hours |66| P3 | Anomaly detected | Ticket, investigate next day |67| P4 | Threshold warning | Dashboard only |6869### Step 6: Correlate Signals70- Link logs to traces via trace_id71- Link metrics to traces via exemplars72- Link alerts to dashboards to runbooks73- Enable drill-down: alert → dashboard → logs → trace7475## Output Format76```markdown77## Observability Design: [System]78### Metrics: [What's collected, where stored]79### Logs: [Format, retention, aggregation]80### Traces: [Sampling, propagation, tooling]81### Dashboards: [List with audiences]82### Alerting: [Severity levels and routing]83```8485## Quality Checklist86- [ ] All three pillars (metrics, logs, traces) addressed87- [ ] Correlation between signals is possible (trace_id)88- [ ] Alerting is based on SLOs, not raw thresholds89- [ ] Dashboards exist for each audience90- [ ] Log format is structured and consistent91- [ ] Sampling strategy balances cost and visibility9293## Edge Cases94- For serverless, use platform-native observability (X-Ray, Cloud Trace)95- For high-cardinality metrics, use histograms not individual timers96- If cost is a concern, aggressive sampling + 100% error capture97- For multi-region, ensure centralized view with region labels
ashutoshsrivastava17