Monitoring

Monitoring Stack

Purpose

Define and enforce monitoring stack configuration with Prometheus, Grafana, Loki, and ELK for metrics, logs, traces, and alerting.

Agent Protocol

Trigger

User request includes: monitoring, prometheus, grafana, loki, elk, elasticsearch, kibana, logstash, beats, filebeat, metricbeat, tempo, jaeger, datasource, dashboard.

Input Context

• Current monitoring setup (if any)
• Infrastructure (Kubernetes, bare metal, cloud)
• Scale (number of nodes/services)
• Budget (open-source vs enterprise)
• Required integrations (Slack, PagerDuty, Opsgenie)

Output Artifact

A markdown document containing:

• Monitoring stack architecture diagram (text)
• Tool selection rationale (Prometheus vs Datadog, etc.)
• Prometheus configuration (scrape configs, recording rules, alerting rules)
• Grafana dashboard structure (folder hierarchy, data source setup)
• Loki configuration (log scrape, labels, retention)
• ELK configuration (index templates, pipeline, shard strategy)
• Integration with on-call (Alertmanager → PagerDuty/Slack)

Response Format

Produce the artifact directly. No preamble, no postamble, no explanations. No filler, no hedging, no transitions. Strip articles a/an/the where unambiguous. Compress output — why use many token when few do trick.

Completion Criteria

• Monitoring architecture defined with data flow
• Prometheus scrape configs for all service types
• Grafana dashboard folder structure defined
• Loki log pipeline with label strategy
• Retention and storage sizing calculated
• Alertmanager routing configured

Max Response Length

4096 tokens

Workflow

Step 1: Select Stack Components

Step 2: Configure Prometheus

Scrape Configs

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

  - job_name: 'node-exporter'
    kubernetes_sd_configs:
      - role: node

  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://app.example.com/health
        - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance

Recording Rules

# rules/recording.yml
groups:
  - name: service_slos
    interval: 30s
    rules:
      - record: service:error_rate_5m
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
      - record: service:latency_p99_5m
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Retention and Sizing

Alerting Rules

# rules/alerts.yml
groups:
  - name: infrastructure
    interval: 30s
    rules:
      - alert: NodeDown
        expr: up{job="node-exporter"} == 0
        for: 1m
        labels:
          severity: P0
        annotations:
          summary: "Node {{ $labels.instance }} down"
          runbook: "https://runbook.example.com/node-down"

      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 10m
        labels:
          severity: P2
        annotations:
          summary: "CPU > 90% on {{ $labels.instance }}"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: P1
        annotations:
          summary: "Disk < 10% free on {{ $labels.instance }}"

      - alert: HighMemoryPressure
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
        for: 5m
        labels:
          severity: P2
        annotations:
          summary: "Memory < 10% available on {{ $labels.instance }}"

  - name: kubernetes
    interval: 30s
    rules:
      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) > 2
        for: 5m
        labels:
          severity: P1
        annotations:
          summary: "Pod {{ $labels.pod }} restarting frequently"

      - alert: PersistentVolumeFilling
        expr: (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) * 100 > 85
        for: 5m
        labels:
          severity: P1
        annotations:
          summary: "PV {{ $labels.persistentvolumeclaim }} > 85% full"

      - alert: PodsPending
        expr: kube_pod_status_phase{phase="Pending"} > 0
        for: 15m
        labels:
          severity: P2
        annotations:
          summary: "{{ $value }} pods pending > 15min"

  - name: applications
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: P1
        annotations:
          summary: "{{ $labels.service }} error rate > 5%"

      - alert: HighLatency
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: P1
        annotations:
          summary: "{{ $labels.service }} p99 latency > 2s"

      - alert: ServiceDown
        expr: probe_success{job="blackbox"} == 0
        for: 1m
        labels:
          severity: P0
        annotations:
          summary: "{{ $labels.target }} is unreachable"

      - alert: ZeroTraffic
        expr: rate(http_requests_total[5m]) == 0
        for: 5m
        labels:
          severity: P0
        annotations:
          summary: "{{ $labels.service }} zero traffic — possible outage"

      - alert: HighLogErrorRate
        expr: sum(rate({job=~".+"} |= "error" [5m])) by (job) > 10
        for: 5m
        labels:
          severity: P2
        annotations:
          summary: "Error log rate > 10/s for {{ $labels.job }}"

Alertmanager Configuration

# alertmanager.yml
route:
  receiver: default
  group_by:
    - alertname
    - severity
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: P0
      receiver: pagerduty-critical
      repeat_interval: 5m
      group_wait: 10s
    - match:
        severity: P1
      receiver: slack-warning
      repeat_interval: 30m
    - match:
        severity: P2
      receiver: slack-info
      repeat_interval: 6h
    - match:
        severity: P3
      receiver: null
    - match_re:
        job: "(node-exporter|kube-state-metrics)"
      receiver: infra-team
      group_by:
        - alertname
        - instance

receivers:
  - name: pagerduty-critical
    pagerduty_configs:
      - routing_key: "${PD_ROUTING_KEY}"
        severity: critical
        description: "{{ .GroupLabels.alertname }} — {{ .CommonAnnotations.summary }}"
        client: "Prometheus"
        client_url: "{{ .ExternalURL }}"

  - name: slack-warning
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: "#alerts-critical"
        title: "{{ .GroupLabels.alertname }}"
        text: "{{ .CommonAnnotations.summary }}"
        color: danger
        fields:
          - title: Severity
            value: "{{ .GroupLabels.severity }}"
          - title: Service
            value: "{{ .GroupLabels.job }}"
          - title: Grafana
            value: "{{ .GeneratorURL }}"

  - name: slack-info
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: "#alerts-info"
        title: "{{ .GroupLabels.alertname }}"
        text: "{{ .CommonAnnotations.summary }}"
        color: warning

  - name: infra-team
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: "#infra-alerts"
        title: "Infra: {{ .GroupLabels.alertname }}"
        text: "{{ .CommonAnnotations.summary }}"

inhibit_rules:
  - source_match:
      severity: P0
    target_match:
      severity: P2
    equal:
      - instance
  - source_match:
      alertname: NodeDown
    target_match:
      alertname: HighCpuUsage
    equal:
      - instance

Step 3: Configure Grafana

Folder Structure

/
├── Infrastructure/
│   ├── Node Exporter / CPU, Memory, Disk, Network
│   └── Kubernetes / Cluster, Nodes, Pods
├── Applications/
│   ├── {service-name} / RED metrics
│   └── Databases / Query latency, connections
├── Business/
│   ├── Orders / Volume, value, funnel
│   └── Users / Registrations, churn, activity
└── SLOs/
    ├── Error Budget / Remaining budget per service
    └── Latency / P50, P95, P99 vs targets

Data Source Configuration

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true
    jsonData:
      timeout: 60
      prometheusType: Prometheus
  - name: Loki
    type: loki
    url: http://loki:3100
    access: proxy
    jsonData:
      timeout: 60
      maxLines: 1000
  - name: Tempo
    type: tempo
    url: http://tempo:3200
    access: proxy
  - name: Elasticsearch
    type: elasticsearch
    url: http://elasticsearch:9200
    database: "[logs-]YYYY.MM.DD"
    jsonData:
      esVersion: 8.0.0
      interval: Daily
      timeField: "@timestamp"

Step 4: Configure Loki

# loki-config.yaml
auth_enabled: false
server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

schema_config:
  configs:
    - from: 2024-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /data/loki/index
    cache_location: /data/loki/index_cache
    shared_store: filesystem
  filesystem:
    directory: /data/loki/chunks

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: 720h  # 30 days
  max_query_series: 500

Log Label Strategy

Step 5: Configure ELK Stack

Filebeat Config

# filebeat.yml
filebeat.inputs:
  - type: container
    paths:
      - /var/log/containers/*.log
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:9200']
  index: "logs-%{+yyyy.MM.dd}"

Index Lifecycle Policy

{
  "policy": {
    "phases": {
      "hot": { "min_age": "0ms", "actions": { "rollover": { "max_size": "50GB", "max_age": "1d" } } },
      "warm": { "min_age": "7d", "actions": { "shrink": { "number_of_shards": 1 } } },
      "cold": { "min_age": "30d", "actions": { "freeze": {} } },
      "delete": { "min_age": "90d", "actions": { "delete": {} } }
    }
  }
}

Step 6: Define Monitoring SLOs

Rules

• Prometheus scrape configs use relabeling for Kubernetes service discovery — never static targets for K8s workloads.
• Grafana dashboards organized by folder hierarchy: Infrastructure → Applications → Business → SLOs. No unorganized dashboards.
• Loki labels limited to low-cardinality values — never use pod or traceID as mandatory labels.
• Alertmanager routes segregated by severity: P0 → PagerDuty, P1 → Slack warning, P2 → Slack info, P3 → suppressed.
• Every Prometheus alert has a runbook annotation pointing to a recovery procedure.
• Retention period set per component: Prometheus 15d local, Loki 30d, ELK 90d with tiered lifecycle.
• Cardinality limit <500,000 active series per Prometheus instance — monitor with prometheus_tsdb_head_series.
• All monitoring components deployed with resource limits and persistent storage.
• SLOs defined with error budget tracking for every production service.
• Alert delivery verified with synthetic tests — never trust alerting without validation.

References

• references/elk-setup.md — ELK Stack Setup Reference
• references/grafana-dashboards.md — Grafana Dashboard Design
• references/loki-setup.md — Loki Setup Reference
• references/monitoring-advanced.md — Monitoring Advanced Topics
• references/monitoring-fundamentals.md — Monitoring Fundamentals
• references/prometheus-setup.md — Prometheus Setup Reference

Handoff

Hand off to management/alerting/SKILL.md for alert rule configuration. Hand off to devops/helm-patterns/SKILL.md for deploying monitoring stack on Kubernetes. Hand off to devops/terraform/SKILL.md for provisioning monitoring infrastructure.