Promql Alerting

PromQL & Alerting

Expert guide to PromQL queries, alert rules, and Grafana dashboard design.

Use this skill when

• Writing PromQL queries for dashboards and alerts
• Designing alert rules with proper thresholds and severity
• Implementing SLO-based monitoring and error budgets
• Building Grafana dashboards with effective visualizations

Do not use this skill when

• Setting up Prometheus infrastructure (see prometheus-configuration)
• Working with non-Prometheus monitoring systems

PromQL Fundamentals

Selectors and Matchers

# Exact match
http_requests_total{method="GET", status="200"}

# Regex match
http_requests_total{method=~"GET|POST"}

# Negative match
http_requests_total{status!="200"}

# Negative regex
http_requests_total{status!~"2.."}

Rate and Increase

# Per-second rate over 5 minutes (for counters)
rate(http_requests_total[5m])

# Increase over time period (total count increase)
increase(http_requests_total[1h])

# Use irate for high-resolution, volatile metrics
irate(http_requests_total[5m])

Aggregations

# Sum across all instances
sum(rate(http_requests_total[5m]))

# Sum by specific labels
sum by (method, status) (rate(http_requests_total[5m]))

# Average, min, max
avg by (instance) (node_cpu_seconds_total{mode="idle"})
max by (job) (up)

# Top 5 by request rate
topk(5, sum by (handler) (rate(http_requests_total[5m])))

# Count of series
count by (status) (http_requests_total)

Common Dashboard Queries

Request Rate

# Total request rate
sum(rate(http_requests_total[5m]))

# Request rate by endpoint
sum by (handler) (rate(http_requests_total[5m]))

# Request rate by status code class
sum by (status_class) (
  label_replace(
    rate(http_requests_total[5m]),
    "status_class", "${1}xx", "status", "(.).*"
  )
)

Error Rate

# Error percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
* 100

# Error rate by service
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (service) (rate(http_requests_total[5m]))
* 100

Latency (Histograms)

# P50 latency
histogram_quantile(0.50,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# P95 latency
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# P99 latency by endpoint
histogram_quantile(0.99,
  sum by (le, handler) (rate(http_request_duration_seconds_bucket[5m]))
)

# Average latency
sum(rate(http_request_duration_seconds_sum[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Resource Utilization

# CPU utilization percentage
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory utilization percentage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Disk utilization percentage
(1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100

# Network throughput
rate(node_network_receive_bytes_total{device="eth0"}[5m]) * 8  # bits/sec

Container Metrics

# Container CPU usage (percentage of requested)
sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))
/
sum by (pod) (kube_pod_container_resource_requests{resource="cpu"})
* 100

# Container memory usage
sum by (pod) (container_memory_working_set_bytes)
/
sum by (pod) (kube_pod_container_resource_limits{resource="memory"})
* 100

# Container restart count
sum by (pod) (increase(kube_pod_container_status_restarts_total[1h]))

Alert Rules

Availability Alerts

groups:
  - name: availability
    rules:
      - alert: ServiceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} on {{ $labels.instance }} is down"

      - alert: HighErrorRate
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (service) (rate(http_requests_total[5m]))
          > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.service }} error rate is {{ $value | humanizePercentage }}"

      - alert: HighLatencyP95
        expr: |
          histogram_quantile(0.95,
            sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
          ) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.service }} P95 latency is {{ $value | humanizeDuration }}"

Resource Alerts

  - name: resources
    rules:
      - alert: HighCPU
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[10m])) * 100) > 85
        for: 10m
        labels:
          severity: warning

      - alert: HighMemory
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 5m
        labels:
          severity: critical

      - alert: DiskWillFillIn24h
        expr: |
          predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Disk on {{ $labels.instance }} predicted to fill within 24 hours"

      - alert: HighDiskUsage
        expr: (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 > 90
        for: 5m
        labels:
          severity: critical

SLO-Based Alerting

# Multi-window multi-burn-rate alerting
# Based on Google SRE practices

  - name: slo
    rules:
      # SLO: 99.9% availability (error budget: 0.1%)
      # Fast burn (2% of budget in 1 hour)
      - alert: SLOErrorBudgetFastBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error budget burning fast — 2% consumed in 1h at current rate"

      # Slow burn (5% of budget in 6 hours)
      - alert: SLOErrorBudgetSlowBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > (6 * 0.001)
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[30m]))
            /
            sum(rate(http_requests_total[30m]))
          ) > (6 * 0.001)
        for: 15m
        labels:
          severity: warning

Recording Rules (Pre-Compute Expensive Queries)

groups:
  - name: request_metrics
    interval: 15s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))

      - record: job:http_error_ratio:rate5m
        expr: job:http_errors:rate5m / job:http_requests:rate5m

      - record: job:http_latency:p95
        expr: |
          histogram_quantile(0.95,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
          )

      - record: job:http_latency:p99
        expr: |
          histogram_quantile(0.99,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
          )

Alert Design Best Practices

1. Page on symptoms, not causes — Alert on "error rate > 5%" not "CPU > 80%".
2. Use for duration — Avoid flapping. for: 5m means condition must hold for 5 minutes.
3. Set meaningful severity — critical = pages someone, warning = investigate next business day.
4. Include runbook links — Add runbook_url annotation with troubleshooting steps.
5. Use recording rules — Pre-compute expensive queries to reduce Prometheus load.
6. Test alert rules — Use promtool test rules with unit tests for alerts.

Common Pitfalls

1. Using irate for alerting — irate is too volatile. Use rate for stable alerting.
2. Missing label in by clause — Forgetting le label in histogram aggregation breaks quantile calculation.
3. Alerting on individual instances — Alert on service-level aggregates, not individual containers.
4. No for duration — Without it, brief spikes trigger alerts.
5. Too many alerts — Alert fatigue leads to ignored alerts. Each alert should be actionable.
6. Fixed thresholds — Use predict_linear() for capacity planning alerts instead of fixed percentages.