# Speckit.sre Observability

> SRE & Observability Specialist - OpenTelemetry distributed tracing, Prometheus metrics, structured logs, and resilience controls.

- Skill: `wedabro/speckit-sre-observability` (Agent Skill)
- Install (CLI): `npx skillmds@latest add wedabro/speckit-sre-observability`
- Raw SKILL.md: https://api.skillmd.com/api/skills/wedabro/speckit-sre-observability/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: wedabro (https://skillmd.com/u/wedabro)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/wedabro/speckit-sre-observability

---


## 🎯 Mission

Design, configure, and enforce enterprise-grade Observability and Site Reliability Engineering (SRE) standards across services. Ensure system state is fully transparent through the Three Pillars (Logs, Metrics, Traces), actionable alerts, and battle-tested resilience patterns.

## 📥 Required Inputs

- `.agents/specs/[feature]/spec.md`, `plan.md`, and `tasks.md`
- Service architecture, topology, and SLA/SLO commitments
- Infrastructure configs (`docker-compose.yml`, Kubernetes manifests, Helm charts)

## 📋 Protocol

### 1. Distributed Tracing & OpenTelemetry (OTel)
- **Trace Context Propagation**: Propagate W3C TraceContext headers (`traceparent`, `tracestate`) across all HTTP, gRPC, and message queue boundaries.
- **Span Granularity**: Instrument high-level operations: incoming HTTP request, external API calls, DB transactions, background jobs, and cache accesses.
- **Baggage & Correlation**: Attach `trace_id`, `span_id`, `service.name`, and `environment` to all telemetry data.

### 2. Metrics Architecture (RED & USE Methods)
- **RED Method (Request-driven Services)**:
  - *Rate*: Requests per second (Counter).
  - *Errors*: Failed requests count partitioned by HTTP status code / error type (Counter).
  - *Duration*: Request latency distribution (Histogram with standardized buckets: `0.005s, 0.01s, 0.025s, 0.05s, 0.1s, 0.25s, 0.5s, 1s, 2.5s, 5s, 10s`).
- **USE Method (Resources - CPU, RAM, Disk, DB Pools)**:
  - *Utilization*: Percent time busy.
  - *Saturation*: Queue length or waiting threads.
  - *Errors*: Device or connection error count.

### 3. Structured Logging Standard
- **Format**: 100% Structured JSON logs emitted to `stdout` / `stderr`.
- **Mandatory Fields**: `timestamp` (ISO 8601 UTC), `level` (`DEBUG`, `INFO`, `WARN`, `ERROR`), `message`, `trace_id`, `span_id`, `service`, `duration_ms`.
- **Zero Sensitive Data**: Automatically scrub passwords, tokens, API keys, card numbers, and PII from log payloads.

### 4. Health Checks & Lifecycle Management
- **Liveness Probe (`/livez`)**: Returns `200 OK` if the process is running. NEVER check downstream dependencies here (prevents cascading restart loops).
- **Readiness Probe (`/readyz`)**: Returns `200 OK` ONLY if database, Redis, and critical downstream dependencies are connected and ready to accept traffic.
- **Graceful Shutdown**: Intercept `SIGTERM`/`SIGINT`, stop accepting new requests, drain existing in-flight connections (15–30s timeout), and close DB/queue connections safely.

### 5. Resilience & Fault Tolerance
- **Circuit Breaker**: Wrap external third-party calls with circuit breakers (e.g. Resilience4j, Tenacity) to fast-fail when error rate exceeds 50%.
- **Rate Limiting**: Enforce Token Bucket / Leaky Bucket rate limits at ingress and per-tenant boundaries.
- **SLO Error Budgets**: Define Service Level Objectives (SLOs) and establish automated alerting when Error Budget consumption rate spikes.

## 📤 Outputs

- OpenTelemetry collector configuration and SDK setup scripts.
- Prometheus scraping configs, dashboards (Grafana JSON), and alert rules.
- Healthcheck endpoints (`/livez`, `/readyz`) and Graceful Shutdown handlers.
- Incident runbooks with triage flowcharts and mitigation procedures.

## 🚫 Guard Rails

- FORBIDDEN: Logging secrets, credentials, auth tokens, or PII in plaintext.
- FORBIDDEN: Checking DB connections in `/livez` liveness probes.
- FORBIDDEN: Running services without graceful shutdown signal trapping.
- FORBIDDEN: Creating metrics with unbounded label cardinality (e.g. user IDs or UUIDs as metric labels).