Prometheus Patterns

Prometheus Patterns

Metric Types

Counter   — monotonically increasing (requests_total, errors_total)
Gauge     — current value, can go up/down (memory_bytes, queue_depth)
Histogram — distribution with buckets (request_duration_seconds)
Summary   — streaming quantiles (less common, prefer histogram)

Naming Convention

<namespace>_<subsystem>_<name>_<unit>
  http_server_requests_total
  http_server_request_duration_seconds
  process_resident_memory_bytes
  job_queue_depth
  db_connections_active

Units: _seconds, _bytes, _total (counter suffix), _ratio (0-1)

prometheus.yml Scrape Config

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: myapp
    static_configs:
      - targets: [myapp:9090]
    metrics_path: /metrics
    scrape_timeout: 10s

  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: $1

Essential PromQL Queries

# Request rate (per second over 5m window)
rate(http_requests_total[5m])

# Error ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))

# p99 latency
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# Memory usage by pod
container_memory_working_set_bytes{namespace="prod"}
  / container_spec_memory_limit_bytes{namespace="prod"}

# CPU throttling ratio
rate(container_cpu_cfs_throttled_seconds_total[5m])
/ rate(container_cpu_cfs_periods_total[5m])

# Dead mans switch (alert if no data)
absent(up{job="myapp"} == 1)

Recording Rules (pre-compute expensive queries)

groups:
  - name: myapp.rules
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_errors:rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          / sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)
          )

Remote Write (to Grafana Cloud / Thanos)

remote_write:
  - url: https://prometheus-prod.grafana.net/api/prom/push
    basic_auth:
      username: ${GRAFANA_CLOUD_USER}
      password: ${GRAFANA_CLOUD_KEY}
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "(job:|node_|container_).*"
        action: keep
    queue_config:
      capacity: 10000
      max_shards: 30
      max_samples_per_send: 5000

Key Rules

• Cardinality kills Prometheus — never use high-cardinality labels (user_id, request_id)
• rate() requires at least 4 data points in the window — use 4× scrape interval minimum
• Use recording rules for any query used in both dashboards and alerts
• irate() for instant spikes, rate() for sustained trends
• Set --storage.tsdb.retention.time=30d and plan disk accordingly (~2 bytes/sample)