# Prometheus Grafana Triage

> Investigate Prometheus, Grafana, and Alertmanager incidents by separating real platform problems from scrape failures and bad rules. Use when alerts fire unexpectedly, targets are down, dashboards look stale, Alertmanager is noisy, or PromQL or rule logic may be wrong.

- Skill: `canepro/prometheus-grafana-triage` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add canepro/prometheus-grafana-triage`
- Raw SKILL.md: https://api.skillmd.com/api/skills/canepro/prometheus-grafana-triage/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: Canepro (https://skillmd.com/u/canepro)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/canepro/prometheus-grafana-triage

---


# Prometheus Grafana Triage

Treat alerting problems as one of three classes: a real platform issue, a broken scrape, or a bad rule. Verify which one it is before touching manifests.

## Use when

- Grafana alerts are firing and the user wants to know why
- Prometheus scrape targets are down
- Alertmanager is noisy or seems stale
- Grafana dashboards disagree with current cluster state
- the user suspects a rule bug or stale metric logic

## Do not use when

- The primary problem is a generic Kubernetes runtime incident with no monitoring angle. Use `k8s-sre-triage`.
- The primary problem is a failing CI job or deployment pipeline. Use the installed GitHub `github:gh-fix-ci` skill for GitHub Actions, or investigate the owning pipeline natively.
- The request is for dashboard design or long-term metrics architecture rather than incident triage.

## Workflow

### 1. Check the live monitoring stack

Verify:
- which cluster actually owns the alert evaluation or scrape path
- the expected state for each cluster, spoke, or optional runtime
- Grafana health
- Prometheus health
- Alertmanager health
- whether you are querying the hub or the spoke agent

Before switching contexts or querying random clusters, identify the source of truth:
- the cluster label on the alert or target
- the Prometheus or Alertmanager instance that is currently evaluating the rule
- whether the data comes from a hub-and-spoke flow or a local Prometheus
- repo docs, runbooks, maintenance schedules, or automation state that say a target is intentionally stopped, parked, retired, or on-demand

Do not assume the dashboard reflects the current truth until Prometheus and its scrape path are confirmed healthy.

### 2. Classify expected state before declaring an incident

For hub-and-spoke or cost-controlled environments, classify every relevant target before interpreting missing metrics or unreachable endpoints:
- `live`: expected to be online now
- `parked`: intentionally shut down, scaled to zero, paused, or on-demand
- `retired`: intentionally removed from service but still visible in old dashboards, labels, or Argo CD apps
- `unknown intent`: no current evidence says whether the target should be online

Use repo-local evidence first:
- README and architecture docs
- operations runbooks
- maintenance or startup/shutdown docs
- GitOps application descriptions
- automation schedules
- recent reports only as secondary context

For a parked or retired target, do not report absent metrics, DNS failure, Argo CD `Unknown`, or `up == 0` as a live platform incident by itself. Report it as expected offline state, stale visibility, or an expected blind spot.
In hub-and-spoke environments, before treating missing hub data as target failure, check remote write and federation flow health first. Confirm whether there is federation lag, and inspect remote write queue depth and backfill status so ingest delay is not mistaken for a real outage.

Escalate an offline target only when:
- the user asked for that target to be checked as online
- a job, migration, maintenance window, or startup runbook expects it to be online
- recent telemetry shows it was online and then dropped unexpectedly
- the shutdown/startup automation itself reports failure

Example: if an AKS spoke is documented as on-demand for cost control, then missing AKS samples and Argo CD `Healthy/Unknown` are expected while it is parked. The SRE finding is the OKE hub health plus the parked-spoke state, not an AKS outage.

### 3. Inspect active alerts

Capture:
- alert name
- state: pending vs firing
- cluster label
- severity
- summary and description

Separate:
- active alert in Alertmanager
- active rule evaluation in Prometheus
- stale visualization in Grafana

For container restart or `OOMKilled` alerts, verify the runtime event separately from the alert freshness:
- current pod `restartCount`
- `lastState.reason`
- `lastState.terminated.exitCode`
- previous container logs when available

Bundled helpers:
- `scripts/alert_summary.py` for active alerts from Prometheus or Alertmanager
- `scripts/prom_target_failures.py` for current scrape failures with `lastError`

### 4. Check scrape health

When `up == 0`, inspect:
- target instance
- job name
- scrape URL
- `lastError`

Confirm the endpoint directly where possible:
- wrong path often returns `404`
- wrong port or bind address often returns `connection refused`
- missing auth returns `401` or `403`

For Prometheus warnings containing `Error on ingesting samples with different value but same timestamp`, use a duplicate-sample workflow before changing scrape config:
- map the warning target IP and port back to a pod, service, endpoint, or
  EndpointSlice
- sample the target directly and count duplicate metric-label-timestamp keys
  when the endpoint is reachable
- distinguish duplicate exposition by the target from duplicate scrape paths or
  overlapping scrape jobs
- inspect the owning source object or GitOps manifest that creates the duplicate
  target data
- after a fix, search Prometheus logs with `--since-time` after the last known
  matching warning; broad log output can include stale historical warnings

### 5. Check the rule logic

Look for common rule problems:
- using `count(metric)` when the correct intent is `sum(metric == 1)`
- alerting on sticky gauges like `last_terminated_reason == 1`
- thresholds that count metric series rather than objects
- rules evaluated against the wrong cluster label

Prefer a corrected query over ad hoc silencing when the rule itself is wrong.
After any PromQL or rule fix, run deterministic rule validation with `promtool check rules` and `promtool test rules` (or equivalent rule tests) before closing the fix loop.

For OOM alerts, avoid treating a sticky `last_terminated_reason` gauge as proof that the problem is still active. A real OOM event can coexist with a stale firing alert.

### 6. Decide the fix path

Pick one:
- runtime issue: hand off to `k8s-sre-triage`
- scrape config issue: fix service, ServiceMonitor, scrape config, port, or path
- rule issue: fix the PromQL and docs
- temporary operational noise: before adding or updating a silence, inspect alertmanager notification policy and route configuration for routing, grouping, repeat interval, and inhibition behavior to ensure the silence maps to the right route scope
- expected offline state: document the parked, retired, or on-demand state and recommend checks only for the next startup or intended-online window

### 7. Verify

After the change:
- target health becomes `up`
- corrected query returns the expected count
- firing or pending state clears as expected
- no important alert coverage was removed accidentally
- changed rules pass deterministic checks such as `promtool check rules` and `promtool test rules`
- expected-state classification is backed by a current repo doc, runbook, automation schedule, or explicit user instruction
- log-derived symptoms have a cutoff proof: the same log query after the last
  observed warning or error timestamp returns no new matches

## Guidance

- Query the system that actually scrapes the target. In a hub-and-spoke design, the spoke agent often holds the real `lastError`.
- Do not trust one layer alone. Compare cluster reality, Prometheus target health, and alert logic.
- Do not collapse "unreachable" into "broken" until expected state is known. On-demand infrastructure often looks broken from dashboards while parked.
- If the rule source lives in a different repo than the workload, say so clearly and patch the right repo.
- `container_memory_working_set_bytes` is supporting evidence, not definitive proof of an exact cgroup kill threshold or kill timestamp. Use it to support an OOM diagnosis, not to overstate one.
- When the evidence proves a real event but not the precise trigger, say so. Distinguish “real incident” from “fully explained incident.”

## DPM and usage analysis

DPM is data points per minute per series: at a 60s scrape interval a series produces 1 DPM, at 15s it produces 4. A DPM spike with flat Active Series usually means a shortened scrape interval or duplicate remote_write senders, not new series. To find what drives it:
- Grafana Cloud billing and usage dashboards break usage down per metric and per source
- `count_over_time(metric[1m])` or `rate()` on suspect metrics shows samples-per-minute directly
- the tsdb status endpoints (see `prometheus-cardinality-troubleshooter`) show which metrics carry the most series

Hand off to `prometheus-cardinality-troubleshooter` when the driver is an active series explosion, and to `prometheus-label-strategy` when the ask is preventing it in instrumentation or scrape config.

## Related specialist skills

Use these for deeper Grafana-specific work after the incident shape is clear:
- Write and fix PromQL directly; use `prometheus-cardinality-troubleshooter` when slowness comes from cardinality.
- `loki` for LogQL, parsers, log-pipeline behavior, and Loki-specific troubleshooting.
- Configure Grafana alert rules, contact points, notification policies, silences, and SLOs directly from Grafana Alerting knowledge.
- `prometheus-cardinality-troubleshooter` for active series explosions, Prometheus OOM, slow queries, ingest limits, or DPM fires.
- `prometheus-label-strategy` for preventing cardinality problems in instrumentation or scrape target labels.
- `loki-label-analyzer` for Loki label strategy and slow log queries caused by bad labels.
- For Grafana MCP setup, follow the upstream grafana/mcp-grafana README directly.

## References

- Read `references/triage-patterns.md` for common scrape and rule failure patterns.

## Scripts

### `scripts/alert_summary.py`

Summarizes active alerts from Prometheus or Alertmanager.

Usage:
```bash
python3 "${CODEX_HOME:-$HOME/.codex}/skills/prometheus-grafana-triage/scripts/alert_summary.py" --prometheus-url http://127.0.0.1:9090
python3 "${CODEX_HOME:-$HOME/.codex}/skills/prometheus-grafana-triage/scripts/alert_summary.py" --alertmanager-url http://127.0.0.1:9093
```

### `scripts/prom_target_failures.py`

Lists non-healthy Prometheus targets with cluster, job, scrape URL, and `lastError`.

Usage:
```bash
python3 "${CODEX_HOME:-$HOME/.codex}/skills/prometheus-grafana-triage/scripts/prom_target_failures.py" --prometheus-url http://127.0.0.1:9090
```