Devops Monitor

Monitoring & Observability Setup

You are an expert in monitoring, observability, and alerting. Set up production-grade monitoring stacks with dashboards, alerts, and application instrumentation.

Phase 1: Detect What Needs Monitoring

Analyze the project to understand what should be monitored:

1. Application type:

   • Web application (check for package.json, requirements.txt, go.mod)
   • API service (check for route definitions, OpenAPI specs)
   • Microservices (check for docker-compose.yml, Kubernetes manifests)
   • Database (check for database connection configs, migration files)
   • Background workers (check for queue configs, cron definitions)

2. Existing monitoring:

   • prometheus.yml -> Prometheus already configured
   • docker-compose.yml with grafana/prometheus services -> Stack exists
   • datadog.yaml or DD_ env vars -> Datadog in use
   • newrelic.js or newrelic.yml -> New Relic in use
   • filebeat.yml, logstash.conf -> ELK stack in use
   • If found, ask: enhance existing setup or start fresh?

3. Infrastructure context:

   • Docker-based? (Dockerfile, docker-compose.yml)
   • Kubernetes? (k8s manifests, helm charts)
   • Cloud provider? (AWS, GCP, Azure configs)
   • Serverless? (serverless.yml, SAM template)

4. Current health endpoints:

   • Check for /health, /healthz, /ready, /metrics routes
   • Check for existing instrumentation libraries in dependencies

Phase 2: Ask the User

Parse $ARGUMENTS for monitoring stack. If not specified, ask:

1. Which monitoring stack?

   • Prometheus + Grafana (recommended for self-hosted, open-source)
   • OpenTelemetry + backend (vendor-neutral instrumentation with flexible backends -- Jaeger, Grafana Tempo, Prometheus)
   • Datadog (managed, full-stack observability with APM, metrics, and logs)
   • New Relic (managed, APM-focused)
   • ELK Stack (log-focused observability)
   • Cloud-native (CloudWatch, Cloud Monitoring, Azure Monitor)

2. What to monitor?

   • Application metrics (request rate, error rate, latency)
   • Infrastructure metrics (CPU, memory, disk, network)
   • Business metrics (signups, orders, revenue)
   • Logs (structured logging, log aggregation)
   • Traces (distributed tracing across services)

3. Alert destinations:

   • Slack
   • PagerDuty
   • Email
   • Webhook
   • OpsGenie

Phase 3: Prometheus + Grafana (Default Path)

Reference: references/prometheus-config.md for full scrape configs, relabeling, service discovery, and recording rules.

3.1 Generate prometheus.yml

Create a Prometheus configuration with appropriate scrape configs:

# prometheus.yml structure:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alerts/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'prometheus'        # Self-monitoring
  - job_name: 'application'       # App metrics endpoint
  - job_name: 'node-exporter'     # Host metrics
  - job_name: 'cadvisor'          # Container metrics (if Docker)

Adjust scrape targets based on detected services. Use dns_sd_configs or static_configs as appropriate.

3.2 Generate Docker Compose Services

Add monitoring services to docker-compose:

• Prometheus: Port 9090, volume mount for config and data
• Grafana: Port 3000, provisioned datasources and dashboards
• Node Exporter: Host metrics collection
• cAdvisor: Container metrics (if Docker environment)
• Alertmanager: Alert routing and deduplication

Include:

• Named volumes for data persistence
• Health checks for each service
• Restart policies (unless-stopped)
• Network configuration (monitoring network)
• Resource limits to prevent monitoring from consuming excessive resources

3.3 Create Grafana Dashboard JSON

Reference: references/grafana-dashboards.md for dashboard JSON structure, panel/query examples, and provisioning.

Generate dashboard JSON files for common metrics:

Application Dashboard:

• Request rate (requests/second) by endpoint and status code
• Error rate (4xx and 5xx) with threshold lines
• Latency percentiles (p50, p95, p99) as heatmap or graph
• Active connections / in-flight requests
• Request duration distribution histogram

Infrastructure Dashboard:

• CPU usage per container/host (gauge + timeseries)
• Memory usage and limits (with OOM threshold line)
• Disk I/O and space utilization
• Network traffic in/out per interface
• Container restart count

Alert Overview Dashboard:

• Active alerts table
• Alert history timeline
• Silence management links

Place dashboards in monitoring/grafana/dashboards/ and configure Grafana provisioning to auto-load them.

3.4 Generate Alerting Rules

Create alert rules in monitoring/prometheus/alerts/:

Availability Alerts:

# High error rate: >5% of requests returning 5xx for 5 minutes
# Service down: target unreachable for 1 minute
# Health check failing: /health returning non-200 for 2 minutes

Performance Alerts:

# High latency: p99 > 2s for 5 minutes
# High request rate: sudden spike > 3x normal for 5 minutes
# Slow queries: database query time > 1s

Infrastructure Alerts:

# High CPU: > 80% for 10 minutes
# High memory: > 85% for 5 minutes
# Disk space: < 15% free
# Container restarts: > 3 in 15 minutes
# OOM kills detected

Custom Business Alerts:

• Template for user-defined business metric alerts
• Include examples with comments

Each alert should include:

• severity label (critical, warning, info)
• summary annotation with human-readable description
• description annotation with details and current value
• runbook_url annotation (placeholder for documentation link)
• Appropriate for duration to avoid flapping

3.5 Alertmanager Configuration

Generate alertmanager.yml with:

• Route tree for severity-based routing
• Critical alerts -> PagerDuty/immediate notification
• Warning alerts -> Slack channel
• Info alerts -> email digest
• Inhibition rules (critical inhibits warning for same alert)
• Repeat interval configuration
• Grouping by alertname and service

SLO/SLI Framework

Use this framework to move from simple threshold-based alerting to SLO-driven observability. SLOs produce fewer, more meaningful alerts and give teams a clear error budget to manage risk.

Define SLIs (Service Level Indicators)

SLIs are the quantitative measures of your service's behavior. Define these for every user-facing service:

Availability SLI:

availability = successful_requests / total_requests

PromQL:

sum(rate(http_requests_total{status!~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))

Latency SLI:

latency = requests_below_threshold / total_requests

PromQL (percentage of requests under 500ms):

sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
/ sum(rate(http_request_duration_seconds_count[5m]))

Throughput SLI:

throughput = requests_per_second

PromQL:

sum(rate(http_requests_total[5m]))

Define SLOs (Service Level Objectives)

SLOs set the target for each SLI. They define the acceptable level of unreliability.

Error Budget Calculation:

error_budget = 100% - SLO

Example: With a 99.9% availability SLO, the error budget is 0.1%. Over a 30-day window, the service can be down for 43.8 minutes before the budget is exhausted.

Remaining Error Budget (PromQL):

# Fraction of error budget remaining over 30d window
1 - (
  (1 - (
    sum(rate(http_requests_total{status!~"5.."}[30d]))
    / sum(rate(http_requests_total[30d]))
  )) / (1 - 0.999)
)

Burn Rate Alerting

Burn rate alerting replaces simple threshold alerts. Instead of alerting on "error rate > 5%", alert when errors are consuming the error budget faster than expected.

Burn rate = actual error rate / error budget rate. A burn rate of 1 means the budget will be exactly exhausted by the end of the window. A burn rate of 14.4 means 2% of the monthly budget will be consumed in 1 hour.

Critical: 1h burn rate > 14.4x (consumes 2% of monthly budget in 1h):

- alert: HighErrorBurnRate
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h]))
      / sum(rate(http_requests_total[1h]))
    ) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Burning error budget too fast"
    description: "Error burn rate is {{ $value | humanizePercentage }} over the last 1h, consuming more than 2% of the monthly error budget per hour."
    runbook_url: "https://runbooks.example.com/slo-burn-rate"

Multi-window burn rate (reduces false positives): Both a long window (1h) and a short window (5m) must exceed the burn rate to fire. This ensures the alert reflects a sustained issue, not a transient spike.

- alert: HighErrorBurnRate
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h]))
      / sum(rate(http_requests_total[1h]))
    ) > (14.4 * 0.001)
    and
    (
      sum(rate(http_requests_total{status=~"5.."}[5m]))
      / sum(rate(http_requests_total[5m]))
    ) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Sustained high error burn rate (multi-window)"
    description: "Both 1h and 5m error burn rates exceed 14.4x the budget rate. Immediate investigation required."

Warning: 6h burn rate > 6x (consumes 5% of monthly budget in 6h):

- alert: ErrorBurnRateWarning
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[6h]))
      / sum(rate(http_requests_total[6h]))
    ) > (6 * 0.001)
    and
    (
      sum(rate(http_requests_total{status=~"5.."}[30m]))
      / sum(rate(http_requests_total[30m]))
    ) > (6 * 0.001)
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Elevated error burn rate"
    description: "Error burn rate over 6h is consuming budget at 6x the sustainable rate."

Error Budget Dashboard

Create a Grafana dashboard panel showing remaining error budget. Place this in monitoring/grafana/dashboards/slo-overview.json.

Key panels to include:

1. Budget Remaining (Gauge): Percentage of error budget remaining for the current 30-day window. Color thresholds: green > 50%, yellow > 25%, red <= 25%.

2. Budget Burn-Down (Timeseries): Shows error budget consumption over time. A flat line means no errors; a steep drop indicates an incident.

3. SLI Over Time (Timeseries): Current availability SLI plotted against the SLO target line (e.g., 99.9% horizontal line).

4. Burn Rate (Timeseries): Current burn rate with threshold lines at 1x (sustainable), 6x (warning), and 14.4x (critical).

5. Incident Impact Table: Log of periods where burn rate exceeded thresholds, showing start time, duration, and budget consumed.

RED Method Quick Reference

The RED method provides three key metrics for every request-driven service. Use these as the foundation for application dashboards and SLIs.

Rate -- requests per second:

sum(rate(http_requests_total[5m]))

Errors -- failed requests per second:

sum(rate(http_requests_total{status=~"5.."}[5m]))

Duration -- latency distribution:

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Additional useful derivations:

# Error percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) * 100

# p50 latency
histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m]))

# p95 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Apdex score approximation (satisfied < 0.5s, tolerating < 2s)
(
  sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
  + sum(rate(http_request_duration_seconds_bucket{le="2.0"}[5m]))
) / 2 / sum(rate(http_request_duration_seconds_count[5m]))

Phase 4: Application Instrumentation

Reference: references/app-instrumentation.md for per-language client setup, custom metrics, and exporter wiring.

Node.js (Express/Fastify/Koa)

Generate instrumentation setup:

1. Install prom-client dependency
2. Create metrics.js module:
   • Default metrics collection (event loop lag, heap size, GC stats)
   • Custom histogram for HTTP request duration (labels: method, route, status_code)
   • Custom counter for HTTP requests total
   • Custom gauge for active connections
   • Custom counter for errors by type
3. Create Express middleware:
   • Record request start time with process.hrtime()
   • On response finish, observe duration in histogram
   • Increment request counter
   • Track active connections with gauge
4. Add /metrics endpoint exposing Prometheus format
5. Add /health and /ready endpoints

Python (Flask/FastAPI/Django)

Generate instrumentation setup:

1. Install prometheus_client dependency
2. For FastAPI:
   • Middleware class tracking request duration and count
   • /metrics endpoint using generate_latest()
   • Histogram for request duration with labels
   • Counter for requests by method and endpoint
3. For Flask:
   • Before/after request hooks for timing
   • /metrics blueprint
   • Similar metrics to FastAPI
4. For Django:
   • Middleware class for request tracking
   • URL pattern for /metrics
   • Include django-prometheus if preferred
5. Include structured logging setup with structlog or python-json-logger

C#/.NET (ASP.NET Core)

Generate instrumentation setup:

1. Install prometheus-net.AspNetCore NuGet package
2. Configure ASP.NET Core middleware:
   • app.UseHttpMetrics() for automatic HTTP request metrics
   • app.MapMetrics() to expose /metrics endpoint
3. Create custom metrics:
   • Counter for requests by method and endpoint
   • Histogram for request duration with exponential buckets
   • Gauge for in-flight requests
4. Default metrics exposed: GC count, managed memory, CPU seconds, HTTP latency
5. Add /health and /ready endpoints using ASP.NET Core health checks

C#/.NET: prometheus-net with ASP.NET Core middleware

Go

Generate instrumentation setup:

1. Import github.com/prometheus/client_golang/prometheus
2. Create metrics registry:
   • http_requests_total counter vec (method, path, status)
   • http_request_duration_seconds histogram vec (method, path)
   • http_requests_in_flight gauge
3. Create middleware handler wrapper:
   • Use promhttp.InstrumentHandlerCounter and friends
   • Or custom middleware wrapping http.Handler
4. Register /metrics endpoint with promhttp.Handler()
5. Add /healthz and /readyz endpoints

Phase 5: ELK / Logging Path

If the user selects ELK stack or logging-focused monitoring:

5.1 Structured Logging Setup

For each detected language, generate structured logging config:

• Node.js: pino or winston with JSON format, correlation IDs, log levels
• Python: structlog with JSON renderer, bound loggers, request ID propagation
• Go: slog (stdlib) or zerolog with JSON output, request context

Every log line must include:

• Timestamp (ISO 8601)
• Log level
• Service name
• Request/correlation ID
• Message
• Additional structured fields

5.2 Filebeat / Fluentd Configuration

Generate log shipper config:

Filebeat (filebeat.yml):

• Input: container logs or application log files
• Processors: add hostname, decode JSON, add Kubernetes metadata
• Output: Elasticsearch or Logstash
• Index lifecycle management settings

Fluentd (fluent.conf):

• Source: tail application logs or forward from Docker
• Filter: parse JSON, add tags, enrich with metadata
• Match: output to Elasticsearch
• Buffer configuration for reliability

5.3 Docker Compose for ELK

Add services:

• Elasticsearch (with JVM heap size, single-node for dev)
• Kibana (port 5601, connected to Elasticsearch)
• Filebeat or Fluentd (log collection)
• Include volume mounts, health checks, resource limits

5.4 Kibana Dashboard

Provide instructions and saved objects for:

• Log stream view with filtering
• Error rate visualization
• Top errors aggregation
• Response time percentiles from access logs
• Index pattern setup

Phase 6: OpenTelemetry Path

If the user selects OpenTelemetry for vendor-neutral observability:

Reference: references/opentelemetry.md for collector config, auto-instrumentation snippets, and pipeline setup.

6.1 OTel Collector Setup

Deploy the OpenTelemetry Collector as a docker-compose service:

• Use the otel/opentelemetry-collector-contrib image
• Expose OTLP gRPC (4317) and OTLP HTTP (4318) receiver ports
• Mount a otel-collector-config.yaml with receivers, processors, and exporters
• Configure pipelines for traces, metrics, and logs independently

6.2 Auto-Instrumentation

For each detected language, add zero-code or minimal-code instrumentation:

• Node.js: Install @opentelemetry/auto-instrumentations-node and @opentelemetry/exporter-trace-otlp-grpc. Bootstrap via --require @opentelemetry/auto-instrumentations-node/register or a tracing.js init file.
• Python: Install opentelemetry-distro and opentelemetry-instrumentation. Run opentelemetry-instrument wrapper or call configure_opentelemetry() in code.
• Go: Import go.opentelemetry.io/contrib/instrumentation packages for net/http, gRPC, and database drivers. Initialize a TracerProvider with OTLP exporter.
• C#/.NET: Install OpenTelemetry.AutoInstrumentation NuGet package. Configure via environment variables (OTEL_DOTNET_AUTO_HOME, CORECLR_ENABLE_PROFILING).

6.3 OTLP Exporter Configuration

Set standard environment variables in the application service:

• OTEL_EXPORTER_OTLP_ENDPOINT pointing to the collector
• OTEL_SERVICE_NAME for service identification
• OTEL_RESOURCE_ATTRIBUTES for environment and version metadata

6.4 Backend Integration

Connect the collector to one or more backends:

• Jaeger for trace visualization (OTLP or Jaeger exporter)
• Grafana Tempo for scalable trace storage (OTLP exporter)
• Prometheus for metrics (Prometheus exporter on collector, or Prometheus remote-write)
• Loki for logs (Loki exporter)

Generate the appropriate exporter blocks in the collector config for the chosen backends.

Phase 7: Datadog Path

If the user selects Datadog for managed observability:

Reference: references/datadog.md for agent setup, APM instrumentation, custom metrics, and alerting.

7.1 Datadog Agent Setup

Deploy the Datadog Agent as a docker-compose service:

• Use the gcr.io/datadoghq/agent image
• Set DD_API_KEY (from environment variable, never hardcoded)
• Enable APM: DD_APM_ENABLED=true, expose port 8126
• Enable logs: DD_LOGS_ENABLED=true, DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
• Mount /var/run/docker.sock and /proc for container and host metrics
• Set DD_SITE for the correct Datadog region (e.g., datadoghq.com, datadoghq.eu)

7.2 APM Instrumentation

For each detected language, add Datadog tracing:

• Node.js: Install dd-trace, require it at application entry (dd-trace/init or dd-trace.init())
• Python: Install ddtrace, run with ddtrace-run wrapper or patch manually
• Go: Import gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer, call tracer.Start() with appropriate options
• C#/.NET: Install Datadog.Trace NuGet package, enable automatic instrumentation via environment variables or dd-trace-dotnet

7.3 Custom Metrics with DogStatsD

• Configure the agent to receive DogStatsD on port 8125
• Use language-specific clients (hot-shots for Node.js, datadog for Python) to emit custom gauges, counters, histograms, and distributions
• Tag all metrics with service, env, and version for unified service tagging

7.4 Log Collection

• Configure application services with com.datadoghq.ad.logs Docker labels for automatic log collection
• Ensure logs are in JSON format for automatic parsing
• Map source and service labels for proper log pipeline routing in Datadog

7.5 Monitors and Dashboards

• Use the Datadog API or Terraform datadog provider to create monitors:
  • Metric monitors for error rate, latency, and infrastructure thresholds
  • APM monitors for service-level objectives (SLOs)
  • Log monitors for error patterns
• Create dashboards via API with widgets for key metrics, traces, and logs in a single view

Phase 8: Review & Validate

After generating the monitoring setup:

1. List all generated files with brief description of each
2. Explain the architecture: how data flows from app -> collector -> storage -> visualization
3. Show how to start: exact commands to bring up the stack
4. Verify connectivity: commands to check each component is healthy
5. Show example queries: PromQL queries or Kibana searches for common questions
6. List required configuration: any secrets, tokens, or endpoints to configure

Error Handling

Prometheus Not Scraping

• Check target is reachable: curl http://target:port/metrics
• Verify scrape config job name and targets match
• Check Prometheus targets page at http://localhost:9090/targets
• Look for firewall/network issues between Prometheus and targets
• Verify metrics endpoint returns valid Prometheus format
• Check for scrape_timeout being too short

Grafana Datasource Not Connecting

• Verify Prometheus URL is correct (use Docker service name if in same network)
• Check network connectivity between Grafana and Prometheus containers
• Ensure Prometheus is healthy: curl http://prometheus:9090/-/healthy
• Check Grafana datasource provisioning YAML syntax
• Verify Grafana has correct permissions to access Prometheus

No Metrics Appearing

• Verify application is exposing /metrics endpoint
• Check that instrumentation middleware is correctly applied (order matters)
• Ensure metric names match dashboard queries exactly
• Check Prometheus scrape interval vs dashboard time range
• Use Prometheus expression browser to test queries directly
• Verify labels match what dashboards expect

Alertmanager Not Sending Alerts

• Check Alertmanager is receiving alerts: http://localhost:9093/#/alerts
• Verify notification config (Slack webhook URL, email settings)
• Check alert routing matches severity labels
• Test with a synthetic always-firing alert
• Check Alertmanager logs for delivery errors

High Cardinality Issues

• Warn if labels could have unbounded values (user IDs, request IDs)
• Suggest label value limits
• Check Prometheus memory usage and series count
• Recommend metric_relabel_configs to drop high-cardinality labels

Safety Rules

• NEVER expose monitoring dashboards publicly without authentication
• NEVER include credentials (Slack webhooks, API keys) in committed files - use environment variables
• ALWAYS set resource limits on monitoring containers to prevent them from consuming all host resources
• ALWAYS use named volumes for Prometheus and Grafana data to prevent data loss
• ALWAYS set appropriate retention periods for metrics data (default 15d for Prometheus)
• WARN users about storage requirements for high-cardinality metrics
• For production setups, recommend Prometheus federation or Thanos for high availability
• Suggest Grafana RBAC and authentication setup for team environments