Design production-ready observability
Instrument services so operators know what is broken, why, and where — before users notice. Three pillars, golden signals, low-noise alerting.
The three pillars
| Pillar |
Purpose |
Key decision |
| Metrics |
Rate, latency, saturation at a glance |
RED method for services; USE method for resources |
| Logs |
Structured event trail with correlation IDs |
JSON, log-level discipline, sample high-volume streams |
| Traces |
End-to-end request flow across services |
Meaningful span boundaries; tail-based sampling for slow/erroring requests |
Golden signals to define first: latency, traffic, errors, saturation — cover these before anything else.
Steps
- Read the lore first.
search_lore for existing observability decisions (dashboards, alert channels, on-call runbooks, APM tooling). Extend in place; don't duplicate.
- Identify the service contract. What does this service promise users? That contract → the SLIs. Route SLO/error-budget design to
slo-architect.
- Design dashboards. Overview → service → component drill-down. Max 7±2 panels per screen; colour semantics (red = critical, amber = warning, green = healthy); SLO target reference lines.
- Define alert thresholds. Prefer symptom-based over cause-based alerts. Require every alert to have: condition, severity, runbook link, on-call owner. Suppress during known maintenance.
- Reduce noise. Deduplicate, set appropriate alert-evaluation windows, distinguish pager alerts (must wake someone) from dashboard-only signals.
- Verify + evidence. Deploy to staging, confirm every golden-signal panel renders with live data; run
record-evidence with test output; submit for review.
Build / Test
- Validate dashboard JSON against the target platform's schema (Grafana, Datadog, CloudWatch) before committing.
- Alert configs: dry-run evaluation against recent telemetry; confirm alert-to-runbook coverage is 1:1.
- Structured logging: emit a test event and confirm all required fields appear in the aggregation layer.
Review checklist
- Golden signals covered — latency/traffic/errors/saturation panels present for every user-facing path.
- Alert hygiene — every alert has severity, condition, and runbook link; no alert fires without a defined owner.
- Noise budget — alert evaluation windows are wide enough to avoid flapping; non-actionable signals are dashboard-only.
- Correlation IDs — all logs carry a request/trace ID so a single request can be followed across services.
- No SLI/SLO work here — SLO targets and error-budget math delegated to
slo-architect.
Rules
- Every alert must have a runbook link before it ships to production.
- Symptom-based alerts (user-perceived latency/error rate) take priority over cause-based (CPU %).
- Do not instrument everything — start with golden signals and add only when a gap causes a missed incident.
Capture lore
The repo's APM tooling, dashboard naming conventions, alert-channel routing, or on-call rotation policy — observability decisions are permanent and cost every future agent a re-search if undocumented. That kind of fact is lore. Capture it via the lore-capture
protocol in your brief (CLAUDE.factory.md, step 11 "Memory contribution"):
call the Memory MCP suggest_lore once at the close of your work — reusable
conventions, gotchas, decisions, and boundaries only, never per-ticket trivia.
1---2name: observability-designer3description: Use when adding observability to a new service, refactoring noisy alerting, or designing a monitoring strategy. Covers the three pillars (metrics/logs/traces), golden-signal dashboards, and alert-noise reduction. For SLO/error-budget math specifically, route to `slo-architect` instead.4---56# Design production-ready observability78Instrument services so operators know what is broken, why, and where — before users notice. Three pillars, golden signals, low-noise alerting.910## The three pillars1112| Pillar | Purpose | Key decision |13|--------|---------|-------------|14| **Metrics** | Rate, latency, saturation at a glance | RED method for services; USE method for resources |15| **Logs** | Structured event trail with correlation IDs | JSON, log-level discipline, sample high-volume streams |16| **Traces** | End-to-end request flow across services | Meaningful span boundaries; tail-based sampling for slow/erroring requests |1718Golden signals to define first: **latency, traffic, errors, saturation** — cover these before anything else.1920## Steps21221. **Read the lore first.** `search_lore` for existing observability decisions (dashboards, alert channels, on-call runbooks, APM tooling). Extend in place; don't duplicate.232. **Identify the service contract.** What does this service promise users? That contract → the SLIs. Route SLO/error-budget design to `slo-architect`.243. **Design dashboards.** Overview → service → component drill-down. Max 7±2 panels per screen; colour semantics (red = critical, amber = warning, green = healthy); SLO target reference lines.254. **Define alert thresholds.** Prefer symptom-based over cause-based alerts. Require every alert to have: condition, severity, runbook link, on-call owner. Suppress during known maintenance.265. **Reduce noise.** Deduplicate, set appropriate alert-evaluation windows, distinguish pager alerts (must wake someone) from dashboard-only signals.276. **Verify + evidence.** Deploy to staging, confirm every golden-signal panel renders with live data; run `record-evidence` with test output; submit for review.2829## Build / Test3031- Validate dashboard JSON against the target platform's schema (Grafana, Datadog, CloudWatch) before committing.32- Alert configs: dry-run evaluation against recent telemetry; confirm alert-to-runbook coverage is 1:1.33- Structured logging: emit a test event and confirm all required fields appear in the aggregation layer.3435## Review checklist3637- **Golden signals covered** — latency/traffic/errors/saturation panels present for every user-facing path.38- **Alert hygiene** — every alert has severity, condition, and runbook link; no alert fires without a defined owner.39- **Noise budget** — alert evaluation windows are wide enough to avoid flapping; non-actionable signals are dashboard-only.40- **Correlation IDs** — all logs carry a request/trace ID so a single request can be followed across services.41- **No SLI/SLO work here** — SLO targets and error-budget math delegated to `slo-architect`.4243## Rules4445- Every alert must have a runbook link before it ships to production.46- Symptom-based alerts (user-perceived latency/error rate) take priority over cause-based (CPU %).47- Do not instrument everything — start with golden signals and add only when a gap causes a missed incident.4849## Capture lore5051**The repo's APM tooling, dashboard naming conventions, alert-channel routing, or on-call rotation policy — observability decisions are permanent and cost every future agent a re-search if undocumented.** That kind of fact is *lore*. Capture it via the **lore-capture52protocol in your brief** (`CLAUDE.factory.md`, step 11 "Memory contribution"):53call the Memory MCP `suggest_lore` once at the close of your work — reusable54conventions, gotchas, decisions, and boundaries only, never per-ticket trivia.
tmj-90