Vendor-neutral OpenTelemetry, SLOs, burn-rate alerts, dashboards and cardinality cost. Use when instrumenting a service, defining SLOs or alerts, or debugging latency or error spikes.
Make systems answer "is it broken, for whom, and why" in minutes. OpenTelemetry first,
backend-agnostic, symptom-based alerting, cost-aware by design.
For GCP data-platform specifics (Composer DAG failures, BigQuery slot/cost spikes, GCS staging,
Cloud Logging MCP calls), use cardume-data:gcp-observability-patterns instead of this skill.
Mapping to Prometheus/Grafana, Datadog, New Relic, Elastic, GCP, AWS; cardinality, sampling, retention, RUM
reference/incident-investigation.md
Investigating an active or past incident from telemetry
1. OpenTelemetry first
OpenTelemetry (OTel) is the default instrumentation layer: one API/SDK per language, one wire
protocol (OTLP), many backends. It keeps the application code free of vendor SDKs.
SDK + auto-instrumentation. Start with zero-code/auto-instrumentation (HTTP servers and
clients, gRPC, DB drivers, messaging). Add manual spans only around business operations
(checkout.place_order) and attributes that answer real questions.
Resource attributes identify the emitter on every signal: service.name,
service.version, deployment.environment.name, plus k8s.*/cloud.* added by resource
detectors or the Collector. Set them via OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES.
Semantic conventions. Use the standard attribute names (http.request.method,
http.route, http.response.status_code, db.system.name, messaging.system,
error.type) so backends and dashboards work without custom mapping. Use the route template
(/orders/{id}), never the raw path.
Context propagation. W3C Trace Context (traceparent, tracestate) across HTTP/gRPC;
inject/extract headers into message metadata for queues (Kafka, SQS, Pub/Sub) or use span
links for batch consumers. baggage only for small, non-sensitive keys.
Collector pipelines. Apps export OTLP to a Collector (agent per node/sidecar and/or a
gateway). Pipelines are receivers → processors → exporters; standard processors:
memory_limiter, resourcedetection/k8sattributes, attributes/transform (OTTL) or
redaction for scrubbing, filter for dropping noise, tail_sampling at the gateway, and
batching (batch processor after any sampling, or exporter-side sending_queue batching). Swapping backends is a
Collector config change, not a code change.
Exemplars attach a trace_id to histogram samples, so a latency spike on a dashboard
links straight to an example trace. Enable them in the SDK and the metrics backend
(Prometheus: --enable-feature=exemplar-storage; spanmetrics: exemplars.enabled).
Semconv stability. HTTP, database, and core resource conventions are stable; GenAI/LLM
conventions (gen_ai.*) are still in development, so pin the semconv version and expect
renames.
2. Three signals (plus profiles), and when to use each
Rule of thumb: alert on metrics, navigate with traces, explain with logs.
3. Structured logging with trace correlation
Emit JSON (or the platform's structured format) with stable keys: timestamp, severity,
message, service.name, trace_id, span_id, plus event-specific fields.
Inject trace_id/span_id from the active span through the logger integration or the OTel
logs bridge; this is what makes "jump from trace to logs" work in every backend.
Log events, not narration: one line per meaningful event, with the error type and the
decision taken. Keep DEBUG off in production by default, switchable at runtime.
Never log secrets or personal data (tokens, passwords, emails, CPF/national IDs, card
numbers, full request bodies): logs are broadly readable and retained, and under LGPD/GDPR a
leak there is a reportable incident. Use field allowlists, redact in the SDK, and redact again
in the Collector as a safety net.
4. Golden signals, RED, USE
Golden signals (services): latency, traffic, errors, saturation.
RED (request-driven services): Rate, Errors, Duration, per route/operation. Measure
duration as a histogram (explicit buckets or exponential/native histograms), never as an
average.
USE (resources: CPU, memory, disks, pools, queues): Utilization, Saturation, Errors.
Saturation (queue depth, thread-pool wait, connection-pool wait, throttling) is usually the
leading indicator.
Queues/async: consumer lag (age of oldest message), processing rate, DLQ rate.
5. SLIs, SLOs, error budgets
Pick user journeys, then SLIs as ratios: good events / valid events (availability:
non-5xx/total; latency: requests under threshold/total; freshness for pipelines).
Set an SLO target and window (e.g. 99.9% over 28-30 days). Error budget = 1 - target
(43 minutes of full outage per 30 days at 99.9%).
Agree on a budget policy: what happens when it is exhausted (freeze risky deploys,
prioritize reliability work).
Alert on burn rate (how fast the budget is consumed), multi-window and multi-burn-rate:
Severity
Burn rate
Long window
Short window
Budget consumed
Page
14.4
1h
5m
2%
Page
6
6h
30m
5%
Ticket
3 (example rules)
1d
2h
10%
Ticket
1
3d
6h
10%
# Recording rule: error ratio over a window (repeat for 5m, 30m, 1h, 2h, 6h, 1d, 3d)
job:slo_errors:ratio_rate1h =
# Prometheus OTLP ingestion maps service.name to the `job` label
sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[1h]))
/ sum by (job) (rate(http_server_request_duration_seconds_count[1h]))
# Fast-burn page for a 99.9% SLO (error budget 0.001)
(job:slo_errors:ratio_rate1h > (14.4 * 0.001)) and (job:slo_errors:ratio_rate5m > (14.4 * 0.001))
Page rows and the 3d/6h row are SRE Workbook Table 5-8; the 1d/2h row comes from its example
rules. The short window makes the alert reset quickly once the problem stops. Latency SLOs use the
same pattern with good = requests under threshold from histogram buckets. Full rule set,
latency variant, and generators (Sloth, Pyrra, OpenSLO): reference/slo-alerting.md.
6. Alert design
Page only on symptoms users feel (SLO burn, total outage, data loss risk). Causes (CPU,
restarts, disk 80%) go to dashboards or ticket-severity alerts unless they predict imminent
user impact (disk full in < 4h).
Every alert is actionable: a human must be able to do something now. If the answer is
"wait and see", it is not a page.
Every alert carries: severity (page/ticket), owner/team, a summary with the impacted
journey and current value, a runbook_url, and links to the dashboard.
Use for: durations or burn-rate windows to avoid flapping; group and inhibit (a region
down should not page 40 times).
Review alerts quarterly: delete or demote any that fired without action.
Changing production routing, receivers, silences, or paging schedules needs explicit
confirmation: a mis-routed alert fails silently until the next outage.
7. Dashboard layout
Per service, top to bottom: (1) SLO status and remaining budget; (2) RED per key route with
p50/p95/p99 from histograms and exemplars; (3) saturation (pools, queues, CPU throttling,
memory); (4) dependencies (outbound RED per downstream); (5) deploy/flag-change annotations.
Use template variables for environment/region/version, consistent units, and keep drill-down
boards separate. Store dashboards as code (JSON/Jsonnet/Terraform) next to the service.
8. Cardinality and cost control
Series count = product of label values. Never use unbounded labels on metrics: user/tenant
IDs (unless a small fixed set), request IDs, raw URLs, error messages, timestamps.
Put high-cardinality detail on spans and logs, which are sampled or indexed selectively.
Drop unused metrics and attributes in the Collector; aggregate before export where possible.
Set retention per signal (e.g. metrics 13 months downsampled, traces 7-14 days, debug logs
days) and log levels per environment.
Estimate the cost impact (new series, GB/day) before shipping a new metric or log stream.
9. Sampling
Head sampling (decided at the root, parentbased_traceidratio): cheap and simple, but
blind to outcome, so it drops most errors at low ratios.
Tail sampling (Collector gateway, after the trace completes): keep 100% of errors and slow
traces, a small percentage of the rest. Needs all spans of a trace routed to the same
Collector (load-balancing exporter by trace ID) and memory for the decision wait.
Compute RED metrics before sampling so sampling never distorts SLIs: either SDK HTTP
semconv metrics (http.server.request.duration) or the Collector spanmetrics connector
(alpha; emits traces.span.metrics.calls/.duration by default, labelled service.name,
span.name, span.kind, status.code). SLO rules must use the names of the source chosen.
10. Backends without lock-in
Instrument with OTel, export OTLP to a Collector, and choose exporters per backend:
Prometheus/Grafana/Loki/Tempo (or Mimir), Datadog, New Relic, Elastic, GCP Cloud Monitoring/
Trace/Logging, AWS CloudWatch/X-Ray. All accept OTLP natively or through a Collector
exporter. Mapping table, query-language equivalents, and migration notes:
reference/backends-and-cost.md. Telemetry stays in the backends the user operates or has
chosen; do not send production data to third-party tools.
11. Frontend / RUM basics
Measure Core Web Vitals (LCP, INP, CLS) and JS errors from real users, propagate
traceparent on fetch/XHR to your own APIs (configure CORS allowlists), and sample sessions.
Mask form inputs and never capture PII in session replay. Details in
reference/backends-and-cost.md.
12. Incident investigation from telemetry
Confirm impact against the SLO: which journey, how bad, since when.
Scope: break down by route, region, version, host/pod, tenant tier, dependency.
Find the change: deploys, flags, config, traffic shifts, dependency incidents.
Follow exemplars from the latency/error panel into traces; find the slow/failing span.
Read correlated logs by trace_id for the exact error.
Mitigate first (rollback, flag off, shed load) via the owner, then root cause.
Close the loop: add the missing signal, alert, or runbook step.
Read-only queries are free; mitigations are changes and go through the user or the owning agent.
Query recipes and a timeline template: reference/incident-investigation.md.
Handover checklist
OTel SDK + resource attributes set; traces propagate across every hop, including queues
Logs structured with trace_id/span_id, no secrets or PII
SLIs as good/valid ratios, SLO target and budget policy agreed with the owner
Burn-rate alerts with severity, owner, runbook link; rules tested
Service dashboard with SLO, RED, saturation, dependencies, deploy annotations
Cardinality, sampling, and retention decisions written down with expected cost
1---2name: observability-patterns3description: "Vendor-neutral OpenTelemetry, SLOs, burn-rate alerts, dashboards and cardinality cost. Use when instrumenting a service, defining SLOs or alerts, or debugging latency or error spikes."4allowed-tools: Read Grep Glob5metadata:6 source: https://skillmd.com/skills/jeffersonvalente/observability-patterns7---89# Observability Patterns1011> Make systems answer "is it broken, for whom, and why" in minutes. OpenTelemetry first,12> backend-agnostic, symptom-based alerting, cost-aware by design.1314For GCP data-platform specifics (Composer DAG failures, BigQuery slot/cost spikes, GCS staging,15Cloud Logging MCP calls), use `cardume-data:gcp-observability-patterns` instead of this skill.1617## Reference files (load only what the task needs)1819| File | Load when |20| :--- | :--- |21| `reference/instrumentation.md` | Adding OTel to Node, Python, Java/Spring, Go; Collector config; log correlation |22| `reference/slo-alerting.md` | Writing SLIs/SLOs, burn-rate rules, alert templates, dashboard layout |23| `reference/backends-and-cost.md` | Mapping to Prometheus/Grafana, Datadog, New Relic, Elastic, GCP, AWS; cardinality, sampling, retention, RUM |24| `reference/incident-investigation.md` | Investigating an active or past incident from telemetry |2526---2728## 1. OpenTelemetry first2930OpenTelemetry (OTel) is the default instrumentation layer: one API/SDK per language, one wire31protocol (OTLP), many backends. It keeps the application code free of vendor SDKs.3233- **SDK + auto-instrumentation.** Start with zero-code/auto-instrumentation (HTTP servers and34 clients, gRPC, DB drivers, messaging). Add manual spans only around business operations35 (`checkout.place_order`) and attributes that answer real questions.36- **Resource attributes** identify the emitter on every signal: `service.name`,37 `service.version`, `deployment.environment.name`, plus `k8s.*`/`cloud.*` added by resource38 detectors or the Collector. Set them via `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES`.39- **Semantic conventions.** Use the standard attribute names (`http.request.method`,40 `http.route`, `http.response.status_code`, `db.system.name`, `messaging.system`,41 `error.type`) so backends and dashboards work without custom mapping. Use the route template42 (`/orders/{id}`), never the raw path.43- **Context propagation.** W3C Trace Context (`traceparent`, `tracestate`) across HTTP/gRPC;44 inject/extract headers into message metadata for queues (Kafka, SQS, Pub/Sub) or use span45 links for batch consumers. `baggage` only for small, non-sensitive keys.46- **Collector pipelines.** Apps export OTLP to a Collector (agent per node/sidecar and/or a47 gateway). Pipelines are `receivers → processors → exporters`; standard processors:48 `memory_limiter`, `resourcedetection`/`k8sattributes`, `attributes`/`transform` (OTTL) or49 `redaction` for scrubbing, `filter` for dropping noise, `tail_sampling` at the gateway, and50 batching (`batch` processor after any sampling, or exporter-side `sending_queue` batching). Swapping backends is a51 Collector config change, not a code change.52- **Exemplars** attach a `trace_id` to histogram samples, so a latency spike on a dashboard53 links straight to an example trace. Enable them in the SDK and the metrics backend54 (Prometheus: `--enable-feature=exemplar-storage`; `spanmetrics`: `exemplars.enabled`).55- **Semconv stability.** HTTP, database, and core resource conventions are stable; GenAI/LLM56 conventions (`gen_ai.*`) are still in development, so pin the semconv version and expect57 renames.5859## 2. Three signals (plus profiles), and when to use each6061| Signal | Answers | Use for | Avoid |62| :--- | :--- | :--- | :--- |63| **Metrics** | How much / how often / how fast, aggregated | SLIs, alerts, dashboards, capacity | Per-user or per-request identity (cardinality) |64| **Traces** | Where did the time/error go in this request | Latency breakdown, dependency failures, fan-out | Long-term aggregates (sampled) |65| **Logs** | What exactly happened, with detail | Errors with context, audit, rare events | Counting things (use metrics) |66| **Profiles** | Which code burns CPU/memory | Hot paths, leaks; correlate via span IDs | First-line alerting |6768OTel Profiles entered public alpha in 2026 (<https://opentelemetry.io/blog/2026/profiles-alpha/>):69usable for evaluation, but expect format and Collector changes before GA.7071Rule of thumb: alert on metrics, navigate with traces, explain with logs.7273## 3. Structured logging with trace correlation7475- Emit JSON (or the platform's structured format) with stable keys: `timestamp`, `severity`,76 `message`, `service.name`, `trace_id`, `span_id`, plus event-specific fields.77- Inject `trace_id`/`span_id` from the active span through the logger integration or the OTel78 logs bridge; this is what makes "jump from trace to logs" work in every backend.79- Log events, not narration: one line per meaningful event, with the error type and the80 decision taken. Keep DEBUG off in production by default, switchable at runtime.81- **Never log secrets or personal data** (tokens, passwords, emails, CPF/national IDs, card82 numbers, full request bodies): logs are broadly readable and retained, and under LGPD/GDPR a83 leak there is a reportable incident. Use field allowlists, redact in the SDK, and redact again84 in the Collector as a safety net.8586## 4. Golden signals, RED, USE8788- **Golden signals (services):** latency, traffic, errors, saturation.89- **RED (request-driven services):** Rate, Errors, Duration, per route/operation. Measure90 duration as a histogram (explicit buckets or exponential/native histograms), never as an91 average.92- **USE (resources: CPU, memory, disks, pools, queues):** Utilization, Saturation, Errors.93 Saturation (queue depth, thread-pool wait, connection-pool wait, throttling) is usually the94 leading indicator.95- Queues/async: consumer lag (age of oldest message), processing rate, DLQ rate.9697## 5. SLIs, SLOs, error budgets98991. Pick user journeys, then SLIs as ratios: `good events / valid events` (availability:100 non-5xx/total; latency: requests under threshold/total; freshness for pipelines).1012. Set an SLO target and window (e.g. 99.9% over 28-30 days). Error budget = `1 - target`102 (43 minutes of full outage per 30 days at 99.9%).1033. Agree on a budget policy: what happens when it is exhausted (freeze risky deploys,104 prioritize reliability work).1054. Alert on **burn rate** (how fast the budget is consumed), multi-window and multi-burn-rate:106107| Severity | Burn rate | Long window | Short window | Budget consumed |108| :--- | :--- | :--- | :--- | :--- |109| Page | 14.4 | 1h | 5m | 2% |110| Page | 6 | 6h | 30m | 5% |111| Ticket | 3 (example rules) | 1d | 2h | 10% |112| Ticket | 1 | 3d | 6h | 10% |113114```promql115# Recording rule: error ratio over a window (repeat for 5m, 30m, 1h, 2h, 6h, 1d, 3d)116job:slo_errors:ratio_rate1h =117 # Prometheus OTLP ingestion maps service.name to the `job` label118sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[1h]))119/ sum by (job) (rate(http_server_request_duration_seconds_count[1h]))120121# Fast-burn page for a 99.9% SLO (error budget 0.001)122(job:slo_errors:ratio_rate1h > (14.4 * 0.001)) and (job:slo_errors:ratio_rate5m > (14.4 * 0.001))123```124125Page rows and the 3d/6h row are SRE Workbook Table 5-8; the 1d/2h row comes from its example126rules. The short window makes the alert reset quickly once the problem stops. Latency SLOs use the127same pattern with `good = requests under threshold` from histogram buckets. Full rule set,128latency variant, and generators (Sloth, Pyrra, OpenSLO): `reference/slo-alerting.md`.129130## 6. Alert design131132- **Page only on symptoms** users feel (SLO burn, total outage, data loss risk). Causes (CPU,133 restarts, disk 80%) go to dashboards or ticket-severity alerts unless they predict imminent134 user impact (disk full in < 4h).135- **Every alert is actionable**: a human must be able to do something now. If the answer is136 "wait and see", it is not a page.137- **Every alert carries**: severity (`page`/`ticket`), owner/team, a summary with the impacted138 journey and current value, a `runbook_url`, and links to the dashboard.139- Use `for:` durations or burn-rate windows to avoid flapping; group and inhibit (a region140 down should not page 40 times).141- Review alerts quarterly: delete or demote any that fired without action.142- **Changing production routing, receivers, silences, or paging schedules needs explicit143 confirmation**: a mis-routed alert fails silently until the next outage.144145## 7. Dashboard layout146147Per service, top to bottom: (1) SLO status and remaining budget; (2) RED per key route with148p50/p95/p99 from histograms and exemplars; (3) saturation (pools, queues, CPU throttling,149memory); (4) dependencies (outbound RED per downstream); (5) deploy/flag-change annotations.150Use template variables for environment/region/version, consistent units, and keep drill-down151boards separate. Store dashboards as code (JSON/Jsonnet/Terraform) next to the service.152153## 8. Cardinality and cost control154155- Series count = product of label values. Never use unbounded labels on metrics: user/tenant156 IDs (unless a small fixed set), request IDs, raw URLs, error messages, timestamps.157- Put high-cardinality detail on spans and logs, which are sampled or indexed selectively.158- Drop unused metrics and attributes in the Collector; aggregate before export where possible.159- Set retention per signal (e.g. metrics 13 months downsampled, traces 7-14 days, debug logs160 days) and log levels per environment.161- Estimate the cost impact (new series, GB/day) before shipping a new metric or log stream.162163## 9. Sampling164165- **Head sampling** (decided at the root, `parentbased_traceidratio`): cheap and simple, but166 blind to outcome, so it drops most errors at low ratios.167- **Tail sampling** (Collector gateway, after the trace completes): keep 100% of errors and slow168 traces, a small percentage of the rest. Needs all spans of a trace routed to the same169 Collector (load-balancing exporter by trace ID) and memory for the decision wait.170- Compute RED metrics before sampling so sampling never distorts SLIs: either SDK HTTP171 semconv metrics (`http.server.request.duration`) or the Collector `spanmetrics` connector172 (alpha; emits `traces.span.metrics.calls`/`.duration` by default, labelled `service.name`,173 `span.name`, `span.kind`, `status.code`). SLO rules must use the names of the source chosen.174175## 10. Backends without lock-in176177Instrument with OTel, export OTLP to a Collector, and choose exporters per backend:178Prometheus/Grafana/Loki/Tempo (or Mimir), Datadog, New Relic, Elastic, GCP Cloud Monitoring/179Trace/Logging, AWS CloudWatch/X-Ray. All accept OTLP natively or through a Collector180exporter. Mapping table, query-language equivalents, and migration notes:181`reference/backends-and-cost.md`. Telemetry stays in the backends the user operates or has182chosen; do not send production data to third-party tools.183184## 11. Frontend / RUM basics185186Measure Core Web Vitals (LCP, INP, CLS) and JS errors from real users, propagate187`traceparent` on fetch/XHR to your own APIs (configure CORS allowlists), and sample sessions.188Mask form inputs and never capture PII in session replay. Details in189`reference/backends-and-cost.md`.190191## 12. Incident investigation from telemetry1921931. **Confirm impact** against the SLO: which journey, how bad, since when.1942. **Scope**: break down by route, region, version, host/pod, tenant tier, dependency.1953. **Find the change**: deploys, flags, config, traffic shifts, dependency incidents.1964. **Follow exemplars** from the latency/error panel into traces; find the slow/failing span.1975. **Read correlated logs** by `trace_id` for the exact error.1986. **Mitigate first** (rollback, flag off, shed load) via the owner, then root cause.1997. **Close the loop**: add the missing signal, alert, or runbook step.200201Read-only queries are free; mitigations are changes and go through the user or the owning agent.202Query recipes and a timeline template: `reference/incident-investigation.md`.203204## Handover checklist205206- [ ] OTel SDK + resource attributes set; traces propagate across every hop, including queues207- [ ] Logs structured with `trace_id`/`span_id`, no secrets or PII208- [ ] SLIs as good/valid ratios, SLO target and budget policy agreed with the owner209- [ ] Burn-rate alerts with severity, owner, runbook link; rules tested210- [ ] Service dashboard with SLO, RED, saturation, dependencies, deploy annotations211- [ ] Cardinality, sampling, and retention decisions written down with expected cost
Run npx get-skill add jeffersonvalente/observability-patterns in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Vendor-neutral OpenTelemetry, SLOs, burn-rate alerts, dashboards and cardinality cost. Use when instrumenting a service, defining SLOs or alerts, or debugging latency or error spikes. It is listed under Coding & Dev Tools on SkillMD.
This skill has not completed SkillMD's automated safety review yet. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.
jeffersonvalente (@jeffersonvalente) published this skill. Their other Agent Skills are listed on their SkillMD profile.