Speckit.sre Observability

🎯 Mission

Design, configure, and enforce enterprise-grade Observability and Site Reliability Engineering (SRE) standards across services. Ensure system state is fully transparent through the Three Pillars (Logs, Metrics, Traces), actionable alerts, and battle-tested resilience patterns.

📥 Required Inputs

• .agents/specs/[feature]/spec.md, plan.md, and tasks.md
• Service architecture, topology, and SLA/SLO commitments
• Infrastructure configs (docker-compose.yml, Kubernetes manifests, Helm charts)

📋 Protocol

1. Distributed Tracing & OpenTelemetry (OTel)

• Trace Context Propagation: Propagate W3C TraceContext headers (traceparent, tracestate) across all HTTP, gRPC, and message queue boundaries.
• Span Granularity: Instrument high-level operations: incoming HTTP request, external API calls, DB transactions, background jobs, and cache accesses.
• Baggage & Correlation: Attach trace_id, span_id, service.name, and environment to all telemetry data.

2. Metrics Architecture (RED & USE Methods)

• RED Method (Request-driven Services):
  • Rate: Requests per second (Counter).
  • Errors: Failed requests count partitioned by HTTP status code / error type (Counter).
  • Duration: Request latency distribution (Histogram with standardized buckets: 0.005s, 0.01s, 0.025s, 0.05s, 0.1s, 0.25s, 0.5s, 1s, 2.5s, 5s, 10s).
• USE Method (Resources - CPU, RAM, Disk, DB Pools):
  • Utilization: Percent time busy.
  • Saturation: Queue length or waiting threads.
  • Errors: Device or connection error count.

3. Structured Logging Standard

• Format: 100% Structured JSON logs emitted to stdout / stderr.
• Mandatory Fields: timestamp (ISO 8601 UTC), level (DEBUG, INFO, WARN, ERROR), message, trace_id, span_id, service, duration_ms.
• Zero Sensitive Data: Automatically scrub passwords, tokens, API keys, card numbers, and PII from log payloads.

4. Health Checks & Lifecycle Management

• Liveness Probe (/livez): Returns 200 OK if the process is running. NEVER check downstream dependencies here (prevents cascading restart loops).
• Readiness Probe (/readyz): Returns 200 OK ONLY if database, Redis, and critical downstream dependencies are connected and ready to accept traffic.
• Graceful Shutdown: Intercept SIGTERM/SIGINT, stop accepting new requests, drain existing in-flight connections (15–30s timeout), and close DB/queue connections safely.

5. Resilience & Fault Tolerance

• Circuit Breaker: Wrap external third-party calls with circuit breakers (e.g. Resilience4j, Tenacity) to fast-fail when error rate exceeds 50%.
• Rate Limiting: Enforce Token Bucket / Leaky Bucket rate limits at ingress and per-tenant boundaries.
• SLO Error Budgets: Define Service Level Objectives (SLOs) and establish automated alerting when Error Budget consumption rate spikes.

📤 Outputs

• OpenTelemetry collector configuration and SDK setup scripts.
• Prometheus scraping configs, dashboards (Grafana JSON), and alert rules.
• Healthcheck endpoints (/livez, /readyz) and Graceful Shutdown handlers.
• Incident runbooks with triage flowcharts and mitigation procedures.

🚫 Guard Rails

• FORBIDDEN: Logging secrets, credentials, auth tokens, or PII in plaintext.
• FORBIDDEN: Checking DB connections in /livez liveness probes.
• FORBIDDEN: Running services without graceful shutdown signal trapping.
• FORBIDDEN: Creating metrics with unbounded label cardinality (e.g. user IDs or UUIDs as metric labels).