Purpose
Make the system operable: what to measure, log, trace, and alert on—with owners and noise control.
When to Use / When NOT to Use
Use when: new services/features; after incidents; alert fatigue; missing dashboards; before deploy of HighRisk paths.
Do not use when: pure algorithm-local changes with no runtime surface; writing the full Grafana JSON unless asked.
Preconditions
Feature/service scope and rough user journey. Prefer CONTEXT_PACK + SLOs if they exist.
Inputs / Outputs
Inputs: scope, IMPACT_REPORT/INCIDENT_LOG optional, existing metrics/alerts if known.
Outputs: OBS_PLAN
Upstream / Downstream
Upstream: delivery-planner, incident-commander, performance-auditor, engineering-os.
Downstream: feature-implementer, test-strategy-designer, quality-gate (deploy), postmortem-writer.
Core Principles
- SLIs from user journeys, not vanity counters.
- Alerts need: signal, threshold, ownership, runbook link, severity.
- Prefer fewer high-quality alerts over many noisy ones.
- Structured logs with correlation IDs; no secrets/PII in log bodies.
- Traces on boundaries (entry, datastore, external calls).
- RED/USE as defaults; adapt to stack.
- Every alert answers “what do I do at 3am?”.
Process
- Journeys / golden signals for the scope.
- SLIs + SLO targets (or explicit “no SLO yet”).
- Metrics list — name, type, labels (cardinality-safe).
- Logging plan — events, fields, redaction.
- Tracing plan — spans, attributes.
- Alerts — condition, sev, routing, runbook outline; explicitly suppress noisy candidates.
- Dashboard minimum viable views.
- Decision.
Evidence Requirements
Tie SLIs to real endpoints/jobs. Cite existing metric names when present.
Stop Conditions / Failure Modes
| Condition | Action |
|---|---|
| Alert with no action/runbook | Revise — do not ship that alert |
| High cardinality label proposal | Block that label |
Severity + Confidence
Missing observability on SEV-1 path → High for deploy-gate conditions.
Output Contract
## OBS_PLAN
Journeys/SLIs/SLOs: ...
Metrics: ...
Logs: ...
Traces: ...
Alerts: ...
Dashboards: ...
Decision: Proceed | ProceedWithConditions | Revise | Block
Handoffs
feature-implementer, quality-gate, incident-commander, performance-auditor.
Never
- Never log secrets or raw PII.
- Never page on symptoms without ownership.
- Never require 50 metrics for a tiny feature.