Tsh Implementing Observability

Observability Patterns

When to Use

• Setting up monitoring and alerting for applications
• Implementing centralized logging
• Adding distributed tracing to microservices
• Designing SLOs/SLIs and error budgets
• Creating dashboards and runbooks

Three Pillars of Observability

Stack Detection

Check which observability stack the project uses:

• prometheus.yml or ServiceMonitor → Prometheus
• fluent-bit.conf or fluentd.conf → Fluent Bit/Fluentd
• otel-collector-config.yaml → OpenTelemetry
• AWS with aws_cloudwatch_* resources → CloudWatch
• datadog-agent or DD_* env vars → Datadog

Use context7 to look up stack-specific configuration syntax.

Solution Decision Matrix

Metrics Stack

Logging Stack

Tracing Stack

Kubernetes Observability Pattern

┌─────────────────────────────────────────────────────┐
│                   Applications                      │
│  (instrumented with OpenTelemetry SDK or auto-inst) │
└──────────────────────┬──────────────────────────────┘
                       │ OTLP
                       ▼
┌─────────────────────────────────────────────────────┐
│            OpenTelemetry Collector                  │
│  (receives, processes, exports telemetry)           │
└───────┬─────────────────┬─────────────────┬─────────┘
        │                 │                 │
        ▼                 ▼                 ▼
   Prometheus          Loki             Tempo/Jaeger
   (metrics)          (logs)            (traces)
        │                 │                 │
        └────────────────┬┴─────────────────┘
                         ▼
                      Grafana
                   (visualization)

SLO/SLI Framework

Key Metrics (RED Method for Services)

Key Metrics (USE Method for Resources)

SLO Definition Template

# Example: API availability SLO
slo:
  name: api-availability
  description: "API returns successful responses"
  sli:
    metric: |
      sum(rate(http_requests_total{status!~"5.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
  target: 99.9%
  window: 30d
  error_budget: 0.1%  # ~43 minutes/month downtime allowed

Alerting Strategy

Alert Severity Levels

Alert Quality Rules

• Actionable: Every alert must have a clear response action
• Relevant: Alert on symptoms (user impact), not causes
• Unique: Avoid duplicate alerts for same incident
• Timely: Alert early enough to prevent impact

Alert Template (Prometheus)

groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 1%)"
          runbook_url: "https://runbooks.example.com/high-error-rate"

Structured Logging

Log Format (JSON)

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "error",
  "message": "Payment processing failed",
  "service": "payment-api",
  "trace_id": "abc123",
  "span_id": "def456",
  "user_id": "user-789",
  "error": {
    "type": "PaymentGatewayError",
    "message": "Connection timeout"
  },
  "context": {
    "payment_id": "pay-123",
    "amount": 99.99
  }
}

Required Log Fields

Process

1. Discover context → Check existing observability setup (Prometheus, CloudWatch, etc.)
2. Choose stack → Use decision matrix based on environment and requirements
3. Instrument apps → Add OpenTelemetry SDK or auto-instrumentation
4. Configure collection → Set up collectors, exporters, and storage
5. Define SLOs → Establish SLIs, targets, and error budgets
6. Create alerts → Implement actionable alerts with runbooks
7. Build dashboards → Create service and infrastructure dashboards
8. Document runbooks → Write response procedures for each alert

Checklist

• All services emit metrics, logs, and traces
• Trace IDs propagated across service boundaries
• Structured logging with consistent format (JSON)
• SLOs defined with error budgets
• Alerts are actionable with runbook links
• Dashboards show service health at a glance
• Log retention policy configured
• PII/sensitive data excluded from logs
• On-call rotation defined for critical alerts

Anti-Patterns

Related Skills

• tsh-implementing-kubernetes - For K8s-native observability setup
• tsh-implementing-ci-cd - For pipeline observability integration
• tsh-managing-secrets - For secure credential storage for observability tools