# Prometheus Patterns

> When to activate: Prometheus, PromQL, metrics, counter, gauge, histogram, summary, scrape, recording rule, remote write, exporter

- Skill: `mattakushi432/prometheus-patterns` (Agent Skill)
- Install (CLI): `npx skillmds@latest add mattakushi432/prometheus-patterns`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mattakushi432/prometheus-patterns/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: Mattakushi432 (https://skillmd.com/u/mattakushi432)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/mattakushi432/prometheus-patterns

---

# Prometheus Patterns

## Metric Types

```
Counter   — monotonically increasing (requests_total, errors_total)
Gauge     — current value, can go up/down (memory_bytes, queue_depth)
Histogram — distribution with buckets (request_duration_seconds)
Summary   — streaming quantiles (less common, prefer histogram)
```

## Naming Convention

```
<namespace>_<subsystem>_<name>_<unit>
  http_server_requests_total
  http_server_request_duration_seconds
  process_resident_memory_bytes
  job_queue_depth
  db_connections_active

Units: _seconds, _bytes, _total (counter suffix), _ratio (0-1)
```

## prometheus.yml Scrape Config

```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: myapp
    static_configs:
      - targets: [myapp:9090]
    metrics_path: /metrics
    scrape_timeout: 10s

  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: $1
```

## Essential PromQL Queries

```promql
# Request rate (per second over 5m window)
rate(http_requests_total[5m])

# Error ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))

# p99 latency
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# Memory usage by pod
container_memory_working_set_bytes{namespace="prod"}
  / container_spec_memory_limit_bytes{namespace="prod"}

# CPU throttling ratio
rate(container_cpu_cfs_throttled_seconds_total[5m])
/ rate(container_cpu_cfs_periods_total[5m])

# Dead mans switch (alert if no data)
absent(up{job="myapp"} == 1)
```

## Recording Rules (pre-compute expensive queries)

```yaml
groups:
  - name: myapp.rules
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_errors:rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          / sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)
          )
```

## Remote Write (to Grafana Cloud / Thanos)

```yaml
remote_write:
  - url: https://prometheus-prod.grafana.net/api/prom/push
    basic_auth:
      username: ${GRAFANA_CLOUD_USER}
      password: ${GRAFANA_CLOUD_KEY}
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "(job:|node_|container_).*"
        action: keep
    queue_config:
      capacity: 10000
      max_shards: 30
      max_samples_per_send: 5000
```

## Key Rules
- Cardinality kills Prometheus — never use high-cardinality labels (user_id, request_id)
- `rate()` requires at least 4 data points in the window — use 4× scrape interval minimum
- Use recording rules for any query used in both dashboards and alerts
- `irate()` for instant spikes, `rate()` for sustained trends
- Set `--storage.tsdb.retention.time=30d` and plan disk accordingly (~2 bytes/sample)