Monitoring Observability

Monitoring & Observability

Build comprehensive observability for your systems.

The Three Pillars

1. Metrics (Prometheus)

# Instrument your app
http_requests_total{method="GET", endpoint="/api/users", status="200"} 1024
http_request_duration_seconds{quantile="0.95"} 0.235

2. Logging (Loki / ELK)

• Structured JSON logging
• Include: timestamp, level, service, trace_id, message
• Centralized log aggregation
• Log levels: DEBUG < INFO < WARN < ERROR < FATAL

3. Tracing (Jaeger / Tempo)

• Trace every request across services
• Span: individual operation with timing
• Distributed context propagation via trace ID headers
• Sample 1-10% of production traffic

Alerting (Alertmanager)

Alert Severity

Golden Signals (Google SRE)

1. Latency: Time to respond
2. Traffic: Requests per second
3. Errors: Error rate (5xx, exceptions)
4. Saturation: Resource utilization

Dashboard Design (Grafana)

Rules

• 3-5 panels per row
• Time series with trend lines, not raw numbers
• RED metrics: Rate, Errors, Duration per service
• USE metrics: Utilization, Saturation, Errors per resource
• Include annotations for deployments, incidents

Best Practices

• Monitor from outside (synthetic checks)
• Monitor everything, alert on what matters
• Use SLOs to define what's "good enough"
• Test alerts (fire drills, tabletop exercises)
• Review dashboards monthly with the team