ffsshhttiikk Observability
What I Do
I provide comprehensive visibility into complex systems by collecting, correlating, and analyzing metrics, logs, and traces. I enable teams to understand system behavior, diagnose issues, and maintain confidence in their infrastructure.
When to Use Me
- Debugging distributed systems
- Understanding system behavior in production
- Proactive monitoring and alerting
- Performance optimization
- Incident investigation and resolution
- Capacity planning
- Service level management
Core Concepts
- Metrics: Quantitative measurements over time
- Logs: Discrete event records
- Traces: End-to-end request tracking
- Dashboards: Visual aggregations of data
- Alerts: Notifications when conditions are met
- MELT: Metrics, Events, Logs, Traces
- Service Level Indicators (SLIs): Reliability measurements
- Service Level Objectives (SLOs): Target reliability levels
- OpenTelemetry: Standard for telemetry data
- Distributed Tracing: Tracking across service boundaries
Code Examples
OpenTelemetry Configuration (YAML):
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['localhost:8888']
processors:
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 1000
spike_limit_mib: 200
resource:
attributes:
- key: service.name
value: my-service
action: upsert
- key: deployment.environment
value: production
action: upsert
exporters:
prometheusremotewrite:
endpoint: https://prometheus.example.com/api/v1/write
tls:
insecure: false
jaeger:
endpoint: jaeger-collector:14250
tls:
insecure: true
loki:
endpoint: https://loki.example.com/loki/api/v1/push
tls:
insecure: false
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch, resource]
exporters: [jaeger]
metrics:
receivers: [prometheus, otlp]
processors: [memory_limiter, batch]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [resource]
exporters: [loki]
Prometheus Metrics (Python):
from prometheus_client import Counter, Histogram, Gauge, Summary
from prometheus_client import start_http_server, REGISTRY
import time
import random
# Custom metrics
REQUEST_COUNT = Counter(
'http_requests_total',
'Total HTTP requests',
['method', 'endpoint', 'status']
)
REQUEST_LATENCY = Histogram(
'http_request_duration_seconds',
'HTTP request latency',
['method', 'endpoint'],
buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)
ACTIVE_USERS = Gauge(
'active_users',
'Number of currently active users',
['region']
)
QUEUE_SIZE = Gauge(
'job_queue_size',
'Size of job processing queue',
['worker_group']
)
BUSINESS_METRICS = Summary(
'orders_processed',
'Number of orders processed',
['product_type']
)
class MetricsMiddleware:
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
start_time = time.time()
def start_response_wrapper(status, headers):
response = self.app(environ, start_response)
duration = time.time() - start_time
method = environ.get('REQUEST_METHOD', 'UNKNOWN')
path = environ.get('PATH_INFO', '/')
status_code = int(status.split()[0])
REQUEST_COUNT.labels(
method=method,
endpoint=path,
status=status_code
).inc()
REQUEST_LATENCY.labels(
method=method,
endpoint=path
).observe(duration)
return response
return self.app(environ, start_response_wrapper)
# Start metrics server
start_http_server(8000)
Structured Logging (JSON):
{
"level": "INFO",
"timestamp": "2024-01-15T10:30:00.123Z",
"service": "payment-api",
"version": "2.3.1",
"environment": "production",
"trace_id": "00-1234567890abcdef-1234567890abcdef-01",
"span_id": "1234567890abcdef",
"event": "payment_processed",
"user_id": "usr_12345",
"order_id": "ord_67890",
"amount": 99.99,
"currency": "USD",
"payment_method": "credit_card",
"duration_ms": 245,
"status": "success",
"metadata": {
"processor": "stripe",
"processor_transaction_id": "tx_abc123",
"retry_count": 0
},
"message": "Payment processed successfully for order ord_67890"
}
Grafana Dashboard (JSON):
{
"dashboard": {
"title": "Service Health Overview",
"tags": ["production", "health"],
"timezone": "browser",
"refresh": "30s",
"panels": [
{
"id": 1,
"title": "Request Rate (RPM)",
"type": "graph",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
"targets": [
{
"expr": "sum(rate(http_requests_total{service=~\"$service\"}[5m])) by (endpoint)",
"legendFormat": "{{ endpoint }}"
}
],
"alert": {
"name": "HighRequestRate",
"conditions": [
{
"evaluator": {
"params": [10000],
"type": "gt"
}
}
],
"executionErrorState": "alerting",
"frequency": "1m"
}
},
{
"id": 2,
"title": "Error Rate (%)",
"type": "gauge",
"gridPos": {"h": 8, "w": 6, "x": 12, "y": 0},
"targets": [
{
"expr": "sum(rate(http_requests_total{status=~\"5..\",service=~\"$service\"}[5m])) / sum(rate(http_requests_total{service=~\"$service\"}[5m])) * 100"
}
],
"fieldConfig": {
"defaults": {
"max": 5,
"thresholds": [
{"color": "green", "value": null},
{"color": "yellow", "value": 1},
{"color": "red", "value": 3}
]
}
}
},
{
"id": 3,
"title": "P95 Latency (ms)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
"targets": [
{
"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[5m])) by (le)) * 1000",
"legendFormat": "P95"
},
{
"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[5m])) by (le)) * 1000",
"legendFormat": "P99"
}
]
},
{
"id": 4,
"title": "Active Traces",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 12, "y": 8},
"targets": [
{
"expr": "sum(traces_spanmetrics_calls_total{service=~\"$service\"})"
}
]
}
],
"templating": {
"list": [
{
"name": "service",
"type": "query",
"datasource": "Prometheus",
"query": "label_values(http_requests_total, service)",
"refresh": 2
}
]
}
}
}
Best Practices
- Implement structured logging - JSON format with consistent fields
- Use OpenTelemetry - Vendor-neutral instrumentation
- Correlate metrics, logs, traces - Understand the full picture
- Define SLOs and alert on them - Measure user-impacting issues
- Create golden signals dashboards - Latency, traffic, errors, saturation
- Retain data strategically - Hot, warm, cold storage tiers
- Use sampling for traces - Manage volume while maintaining visibility
- Label everything consistently - Service, environment, region, team
- Test your alerts - Don't discover issues during incidents
- Automate remediation - Self-healing for known failure modes