Observability Instrumenter
Add the three pillars — metrics, logs, traces — so a service is debuggable in production. Instrument
what answers real questions; don't bury signal under noise.
When to use
- "Add metrics/logging/tracing" / "instrument this service"
- "Set up Prometheus / Grafana / OpenTelemetry"
- "I have no visibility into what's happening in prod"
Metrics — use the RED method
For every request path expose:
- Rate — requests per second (counter, labelled by route/method).
- Errors — error count/ratio (counter, labelled by route/status class).
- Duration — latency histogram (so you can compute P50/P95/P99).
For consumers/queues add lag, processed, retried, dead-lettered. For datastores add pool
utilization and query duration. Expose /metrics for Prometheus. Avoid high-cardinality labels
(no user ids, no raw paths with ids) — they blow up Prometheus.
Logs — structured + correlated
- JSON structured logs in production; include
request_id/correlation_id, route, status, latency.
- Propagate the correlation id from inbound request → downstream calls → Kafka headers, so one
request can be traced across services by id.
- Log levels with intent: error = needs attention; warn = degraded; info = lifecycle; debug = dev only.
- Never log secrets, tokens, full PII, or card/bank numbers.
Traces — OpenTelemetry
- Auto-instrument HTTP/gRPC server + client; add spans around DB calls, external vendors, and Kafka
produce/consume. Propagate trace context via W3C
traceparent (and into Kafka headers).
- Export to an OTLP collector → Tempo/Jaeger. Sample sensibly (e.g. tail-based or a ratio) to control cost.
SLIs & alerts to recommend
- Availability (success ratio), latency (P99), error rate, saturation (CPU/mem/pool), consumer lag.
- Suggest alert thresholds tied to an SLO (e.g. P99 latency > target for 5m; error rate > 1% for 5m;
consumer lag growing for 10m).
Steps
- Identify language/framework and key paths (HTTP routes, consumers, DB, external vendors).
- Wire metrics (RED), structured logging with correlation id, and OTel tracing with propagation.
- Provide a starter Grafana dashboard outline and the alert rules above.
- Verify
/metrics scrapes and a trace appears end-to-end before finishing.
Pairs with an observability-bootstrap compose stack (Prometheus/Grafana/Loki/Tempo) for local testing.
1---2name: observability-instrumenter3description: Use when a service lacks proper metrics, logs, or traces and you want to add the three pillars of observability without over-instrumenting. Adds Prometheus metrics (RED method - rate, errors, duration), structured logging with correlation/request ids, and OpenTelemetry distributed tracing with context propagation across service and Kafka boundaries. Recommends the key SLI metrics and alert thresholds to watch. Trigger when the user asks to add metrics/logging/tracing, instrument a service, set up Prometheus/Grafana/OpenTelemetry, or debug why they have no visibility into production.4license: MIT5---67# Observability Instrumenter89Add the three pillars — metrics, logs, traces — so a service is debuggable in production. Instrument10what answers real questions; don't bury signal under noise.1112## When to use13- "Add metrics/logging/tracing" / "instrument this service"14- "Set up Prometheus / Grafana / OpenTelemetry"15- "I have no visibility into what's happening in prod"1617## Metrics — use the RED method18For every request path expose:19- **Rate** — requests per second (counter, labelled by route/method).20- **Errors** — error count/ratio (counter, labelled by route/status class).21- **Duration** — latency histogram (so you can compute P50/P95/P99).2223For consumers/queues add **lag**, processed, retried, dead-lettered. For datastores add pool24utilization and query duration. Expose `/metrics` for Prometheus. Avoid high-cardinality labels25(no user ids, no raw paths with ids) — they blow up Prometheus.2627## Logs — structured + correlated28- JSON structured logs in production; include `request_id`/`correlation_id`, route, status, latency.29- Propagate the correlation id from inbound request → downstream calls → Kafka headers, so one30 request can be traced across services by id.31- Log levels with intent: error = needs attention; warn = degraded; info = lifecycle; debug = dev only.32- Never log secrets, tokens, full PII, or card/bank numbers.3334## Traces — OpenTelemetry35- Auto-instrument HTTP/gRPC server + client; add spans around DB calls, external vendors, and Kafka36 produce/consume. Propagate trace context via W3C `traceparent` (and into Kafka headers).37- Export to an OTLP collector → Tempo/Jaeger. Sample sensibly (e.g. tail-based or a ratio) to control cost.3839## SLIs & alerts to recommend40- Availability (success ratio), latency (P99), error rate, saturation (CPU/mem/pool), consumer lag.41- Suggest alert thresholds tied to an SLO (e.g. P99 latency > target for 5m; error rate > 1% for 5m;42 consumer lag growing for 10m).4344## Steps451. Identify language/framework and key paths (HTTP routes, consumers, DB, external vendors).462. Wire metrics (RED), structured logging with correlation id, and OTel tracing with propagation.473. Provide a starter Grafana dashboard outline and the alert rules above.484. Verify `/metrics` scrapes and a trace appears end-to-end before finishing.4950Pairs with an `observability-bootstrap` compose stack (Prometheus/Grafana/Loki/Tempo) for local testing.
Shravan-Amberkar