Verify an Alert's Claim Before Replying
Overview
An alert message is a claim, not a fact. It reflects one query's result at one moment, possibly minutes-to-hours ago, possibly from a rule with a known artifact. Answering from the alert text alone propagates whatever it got wrong. The loop: re-derive the claim from live data, classify it, then draft a reply.
The loop
- Extract the claim. What exactly does the alert assert? Which entity (node, service, cluster), which condition, at what time? Pull these from the alert's labels/annotations, not from the human's paraphrase.
- Find the originating rule. Get the rule's actual query — not the alert summary. The query defines what "down" meant to the rule. (Grafana-managed: the rule UID is in the alert's labels; fetch it via
/api/v1/provisioning/alert-rules/<uid> — see the grafana-alert-rule-api skill.)
- Re-run the same query twice: at the alert's firing timestamp (
&time=<then>) and now. This splits the world:
- fired-then and true-now → ongoing, true positive
- fired-then but false-now → was real, since recovered (stale alert / already resolved)
- can't reproduce at fire-time → suspect the rule (wrong labels, scrape gap, threshold artifact)
- Cross-check with an independent source. The metric and the alert share a pipeline; confirm through a different one —
kubectl get nodes / service health endpoint / logs. Metric says NotReady but kubectl says Ready ⇒ telemetry problem, not an outage.
- Classify explicitly in the reply — four verdicts:
- true positive (ongoing)
- true-but-recovered
- false positive — only once you've named the mechanism (stale data, known artifact, mislabeled series); "can't reproduce" alone is suspicion, not a verdict
- telemetry fault — the query reproduces but the independent check disagrees (step 4): the pipeline is lying, the system is fine, and the fix is in monitoring, not the system
- Draft, don't post. Write the reply and show it to the human for review before it goes anywhere. Verification earns confidence, not authority to speak for someone.
Reply shape
One sentence of verdict, then evidence with timestamps:
Real but recovered: node X was NotReady 14:02–14:31 UTC (kubelet down, matches the alert window); it's Ready now and the alert resolved at 14:33. No action needed.
Include the query or command used, so the reader can re-check.
Common mistakes
- Answering from the alert text or a dashboard screenshot without re-querying.
- Querying only "now" — a recovered alert looks like a false positive unless you also query fire-time.
- Trusting the same telemetry pipeline that produced the alert as its own confirmation.
- Saying "false positive" without naming the mechanism — if you can't say why it misfired, you haven't verified, you've guessed.
- Posting the reply directly instead of drafting for the human.
1---2name: verify-alert-before-replying3description: Use when someone pastes an alert, a monitoring digest, or a claim like "X is down / is this real?" and expects an answer — before replying, agreeing, or acting on it. Also when triaging whether a firing alert is a true positive, stale, or a false positive4---56# Verify an Alert's Claim Before Replying78## Overview910An alert message is a claim, not a fact. It reflects one query's result at one moment, possibly minutes-to-hours ago, possibly from a rule with a known artifact. Answering from the alert text alone propagates whatever it got wrong. The loop: re-derive the claim from live data, classify it, then draft a reply.1112## The loop13141. **Extract the claim.** What exactly does the alert assert? Which entity (node, service, cluster), which condition, at what time? Pull these from the alert's labels/annotations, not from the human's paraphrase.152. **Find the originating rule.** Get the rule's actual query — not the alert summary. The query defines what "down" meant to the rule. (Grafana-managed: the rule UID is in the alert's labels; fetch it via `/api/v1/provisioning/alert-rules/<uid>` — see the grafana-alert-rule-api skill.)163. **Re-run the same query twice**: at the alert's firing timestamp (`&time=<then>`) and now. This splits the world:17 - fired-then and true-now → **ongoing, true positive**18 - fired-then but false-now → **was real, since recovered (stale alert / already resolved)**19 - can't reproduce at fire-time → **suspect the rule** (wrong labels, scrape gap, threshold artifact)204. **Cross-check with an independent source.** The metric and the alert share a pipeline; confirm through a different one — `kubectl get nodes` / service health endpoint / logs. Metric says NotReady but kubectl says Ready ⇒ telemetry problem, not an outage.215. **Classify explicitly** in the reply — four verdicts:22 - **true positive** (ongoing)23 - **true-but-recovered**24 - **false positive** — only once you've named the mechanism (stale data, known artifact, mislabeled series); "can't reproduce" alone is suspicion, not a verdict25 - **telemetry fault** — the query reproduces but the independent check disagrees (step 4): the pipeline is lying, the system is fine, and the fix is in monitoring, not the system266. **Draft, don't post.** Write the reply and show it to the human for review before it goes anywhere. Verification earns confidence, not authority to speak for someone.2728## Reply shape2930One sentence of verdict, then evidence with timestamps:3132> Real but recovered: node X was NotReady 14:02–14:31 UTC (kubelet down, matches the alert window); it's Ready now and the alert resolved at 14:33. No action needed.3334Include the query or command used, so the reader can re-check.3536## Common mistakes3738- Answering from the alert text or a dashboard screenshot without re-querying.39- Querying only "now" — a recovered alert looks like a false positive unless you also query fire-time.40- Trusting the same telemetry pipeline that produced the alert as its own confirmation.41- Saying "false positive" without naming the mechanism — if you can't say *why* it misfired, you haven't verified, you've guessed.42- Posting the reply directly instead of drafting for the human.
gengwg