# Alert Hygiene

> Use when facing flapping alerts, alert fatigue, recurring noisy incidents, threshold audits, or SLO-alert redesign questions for a GCP monitoring project.

- Skill: `dkpapadopoulos/alert-hygiene` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add dkpapadopoulos/alert-hygiene`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dkpapadopoulos/alert-hygiene/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: dkpapadopoulos (https://skillmd.com/u/dkpapadopoulos)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/dkpapadopoulos/alert-hygiene

---


# Alert Hygiene Analysis

Use when facing alert noise, flapping alerts, alert fatigue, threshold audits, or SLO-alert redesign questions. Analyzes alert policies and incident history for a single GCP monitoring project. Produces an action-class prescriptive report: Do Now items (PR-ready config changes with strict gating), Investigate items (bounded discovery with two-stage DoD), and Needs Decision items (strategy/policy choices with named owners). Runs straight through from data pull to final report.

**When NOT to use:**
- Active incident investigation — use incident-analysis instead (tiered log investigation with mitigation playbooks)
- SLO design from scratch — this skill identifies SLO-redesign *candidates*, it does not design SLOs or generate burn-rate PromQL
- Multi-project analysis — v1 is scoped to one monitoring project per invocation

## Behavioral Constraints

### 1. Scope Restriction

All queries are scoped to one monitoring project per invocation. The user must provide the monitoring project ID before analysis begins. If not provided, ask once.

### 2. Temp-File Execution Pattern

All bulk data operations write to session-scoped temp files via `mktemp`. This keeps large JSON payloads out of conversation context and makes the shell path debuggable:

```bash
WORK_DIR=$(mktemp -d /tmp/ah-XXXXXX)
```

All scripts read from and write to `$WORK_DIR`. Clean up at end of session.

### 3. No Checkpoint Gates

The analysis runs straight through from data pull to final report. The frequency table appears in the report appendix, not as a separate pause point. Do not ask the user for input mid-analysis.

### 4. Confidence Inline Flags

When an inference has less than high confidence (e.g., inferring a nightly batch from time-of-day concentration), state the inference and flag it inline: *"Inferred nightly batch from 100% h02 concentration — verify before applying mute window."* Do not gate the report on user confirmation.

## Tier Detection

Run the shared observability preflight before data access:

```bash
bash "$(dirname "$0")/../scripts/obs-preflight.sh"
```

If `gcloud` is `unauthenticated`, run `gcloud auth login` before proceeding. If `gcloud` is `missing`, fall to Tier 3.

### Tier 1 — REST API via scripts (default for inventory and bulk)

Default path for all data pull. Uses `gcloud auth print-access-token` for auth, `curl` for requests, Python scripts for pagination and extraction. MCP truncates at ~100K characters and cannot handle monitoring projects with >50 policies — REST has no such limits.

```bash
WORK_DIR=$(mktemp -d /tmp/ah-XXXXXX)
SCRIPTS_DIR="$(dirname "$0")/../skills/alert-hygiene/scripts"

python3 "$SCRIPTS_DIR/pull-policies.py" \
    --project "$MONITORING_PROJECT" \
    --output "$WORK_DIR/policies.json"

python3 "$SCRIPTS_DIR/pull-incidents.py" \
    --project "$MONITORING_PROJECT" \
    --days 14 \
    --output "$WORK_DIR/alerts.json"

python3 "$SCRIPTS_DIR/compute-clusters.py" \
    --policies "$WORK_DIR/policies.json" \
    --alerts "$WORK_DIR/alerts.json" \
    --output "$WORK_DIR/clusters.json"
```

### Tier 2 — MCP (targeted enrichment only)

If `list_time_series` or `list_alert_policies` MCP tools are available, use them **only** for targeted enrichment after cluster stats are computed:
- Single policy detail lookups (reading full PromQL query text)
- Time-series queries for metric validation on nominated clusters (< 50 results expected)

Never use MCP for bulk inventory pulls or incident history.

### Tier 3 — Guidance only

If neither gcloud nor MCP tools are available, provide manual Cloud Console instructions.

## Analysis Flow

### Stage 0: Validate Data Access

Before any analysis, discover and validate the monitoring project:

1. If the user provides a project ID, verify it is a **monitoring project** (has alert policies), not just a resource project. Run: `python3 pull-policies.py --project PROJECT --output /tmp/ah-check.json`.
2. **If zero policies returned:** The project is likely a resource project scoped under a different monitoring project. Provide actionable guidance: *"Project {X} returned 0 alert policies. This is likely a resource project, not the monitoring project. Try: (a) project ending in `-monitoring` (e.g., `{org}-monitoring`), (b) check Cloud Console > Monitoring > Settings > Metrics Scope to find which project hosts the alert policies, (c) run `gcloud alpha monitoring policies list --project={candidate}` to test candidates."* Do not just ask the user — give them the diagnostic path.
3. If the user does not provide a project, check if recent incident context or prior conversation names it. If not, ask once with the guidance above.
4. Verify incidents return non-empty: run pull-incidents.py with `--days 1` as a quick check. If zero incidents but policies exist, the window may be too narrow or incidents are in a different project — warn with specifics.
5. If both return data: `"Monitoring project: {project}, {N} policies found, incidents available. Proceeding with {days}-day analysis."`
6. **GitHub org and IaC repos for links:** Check if recent conversation or CLAUDE.md names a GitHub org and IaC repos. If not, ask once: *"Which GitHub org hosts your monitoring IaC, and which repo(s)? (e.g., `example-org`, repos `monitoring` and `k8s`). Used for IaC links in the report — skip if not applicable."* Store as `{github_org}` and `{iac_repos}` (list). If the user skips, omit IaC links.

Fail early — do not proceed to Stage 1 with an empty or wrong project.

### Stage 1: Pull Data

Run pull-policies.py and pull-incidents.py with the validated project. Verify non-empty results. Report counts:
- Total policies (enabled/disabled)
- Total incidents in window
- Policies by squad label

#### Optional: SLO Config Enrichment

After pulling policies and incidents, attempt to fetch SLO service definitions from GitHub. This is optional enrichment — the core analysis runs without it.

**Preconditions:** `{github_org}` was provided in Stage 0 AND `gh` CLI is available AND authenticated (`gh auth status` succeeds). If any precondition fails, write fallback artifacts and skip.

**Flow:**

```bash
REPO="${github_org}/monitoring"

# Short-circuit if preconditions not met
if [ -z "${github_org:-}" ]; then
  echo '{"status":"unavailable","reason":"github_org_not_provided","count":0}' \
    > "$WORK_DIR/slo-source-status.json"
  echo '[]' > "$WORK_DIR/slo-services.json"
elif ! command -v gh &>/dev/null || ! gh auth status &>/dev/null 2>&1; then
  echo '{"status":"unavailable","reason":"gh_not_available","count":0}' \
    > "$WORK_DIR/slo-source-status.json"
  echo '[]' > "$WORK_DIR/slo-services.json"
else
  # Step 1: fetch (checked)
  if ! gh api "repos/$REPO/contents/tf/slo-config.yaml" \
       --jq '.content' > "$WORK_DIR/slo-raw-b64.txt" 2>"$WORK_DIR/slo-fetch-err.txt"; then
    echo '{"status":"unavailable","reason":"fetch_failed","count":0}' \
      > "$WORK_DIR/slo-source-status.json"
    echo '[]' > "$WORK_DIR/slo-services.json"
  # Step 2: decode (checked)
  elif ! base64 -d < "$WORK_DIR/slo-raw-b64.txt" > "$WORK_DIR/slo-config.yaml" 2>/dev/null; then
    echo '{"status":"unavailable","reason":"decode_failed","count":0}' \
      > "$WORK_DIR/slo-source-status.json"
    echo '[]' > "$WORK_DIR/slo-services.json"
  # Step 3: extract service names (checked — Ruby stdlib YAML, no PyYAML)
  elif ! ruby -ryaml -rjson -e '
    cfg = YAML.safe_load(File.read(ARGV[0])) || {}
    services = (cfg["services"] || []).map { |s| s["name"] }.compact
      .map { |n| n.downcase.gsub(/[_.]/, "-").gsub(/-(prod|staging|pta)$/, "") }
    File.write(ARGV[1], JSON.generate(services))
    File.write(ARGV[2], JSON.generate({
      status: services.empty? ? "empty" : "ok",
      count: services.length
    }))
  ' "$WORK_DIR/slo-config.yaml" "$WORK_DIR/slo-services.json" \
    "$WORK_DIR/slo-source-status.json" 2>"$WORK_DIR/slo-parse-err.txt"; then
    echo '{"status":"unavailable","reason":"parse_failed","count":0}' \
      > "$WORK_DIR/slo-source-status.json"
    echo '[]' > "$WORK_DIR/slo-services.json"
  fi
  rm -f "$WORK_DIR/slo-raw-b64.txt"
fi
```

**On-disk artifacts (always written, every branch):**
- `slo-services.json` — normalized service name list or `[]`
- `slo-source-status.json` — `{"status": "ok|empty|unavailable", "reason": "...", "count": N}`

Names in `slo-services.json` are normalized with the same rules as `service_key` in compute-clusters.py: lowercase, strip environment suffixes (`-prod`, `-staging`, `-pta`), replace separators (`_`, `.`) with `-`.

**Stage 1 report line addition:** `"{M} services with SLO definitions"` when status is `ok`, `"SLO config: {reason}"` otherwise.

#### Optional: IaC Module Discovery

After SLO enrichment, build a `display_name → file_path` mapping by walking the IaC repo tree via `gh api`. This replaces the previous `gh search code` approach, which returned zero results because GCP-generated policy IDs never appear in Terraform source.

**Preconditions:** `{github_org}` and `{iac_repos}` were provided in Stage 0 AND `gh` is available AND authenticated. If any fail, write fallback and skip.

**Why repo tree walk instead of `gh search code`:**
- Policy IDs are assigned at `terraform apply` time — they do not exist in `.tf` files, so searching for them always returns 0 results
- `gh search code` has a 10 req/min rate limit and frequently hits 403 on private orgs
- `gh api` (used here) has a 5,000 req/hour limit and works reliably on private repos
- The results are deterministic: if a `display_name` matches, the file path is confirmed

**Flow:**

For each repo in `{iac_repos}` that contains a `tf/` directory:

```bash
REPO="${github_org}/${iac_repo}"
IAC_DIR="$WORK_DIR/iac-discovery"
mkdir -p "$IAC_DIR"

# Step 1: List module directories
gh api "repos/$REPO/contents/tf/modules" --jq '.[].name' \
  > "$IAC_DIR/module-names.txt" 2>/dev/null

# Step 2: For each module, fetch the main TF file and extract display_name values
python3 -c "
import subprocess, json, re, os
repo = '$REPO'
iac_dir = '$IAC_DIR'
modules = open(f'{iac_dir}/module-names.txt').read().strip().split('\n')
mapping = {}
for mod in modules:
    # Try common TF file names in the module
    for fname in ['main.tf', 'google_monitoring_alert_policy.tf', f'{mod}.tf']:
        try:
            result = subprocess.run(
                ['gh', 'api', f'repos/{repo}/contents/tf/modules/{mod}/{fname}',
                 '--jq', '.content'],
                capture_output=True, text=True, timeout=10)
            if result.returncode != 0:
                continue
            import base64
            content = base64.b64decode(result.stdout.strip()).decode('utf-8')
            # Extract display_name values from TF
            for m in re.finditer(r'display_name\s*=\s*\"([^\"]+)\"', content):
                dn = m.group(1)
                # Skip condition display names (inside conditions {} blocks)
                # Only keep top-level policy display_name
                mapping[dn] = {
                    'repo': repo,
                    'module': mod,
                    'file': f'tf/modules/{mod}/{fname}',
                    'type': 'module_definition'
                }
            break  # Found file, stop trying alternatives
        except Exception:
            continue

# Step 3: Also scan squad directories for module invocations
try:
    result = subprocess.run(
        ['gh', 'api', f'repos/{repo}/contents/tf/squads', '--jq', '.[].name'],
        capture_output=True, text=True, timeout=10)
    squads = result.stdout.strip().split('\n') if result.returncode == 0 else []
except Exception:
    squads = []

for squad in squads:
    try:
        result = subprocess.run(
            ['gh', 'api', f'repos/{repo}/contents/tf/squads/{squad}',
             '--jq', '.[].name'],
            capture_output=True, text=True, timeout=10)
        files = [f for f in result.stdout.strip().split('\n')
                 if f.endswith('.tf')]
    except Exception:
        continue
    for fname in files:
        try:
            result = subprocess.run(
                ['gh', 'api',
                 f'repos/{repo}/contents/tf/squads/{squad}/{fname}',
                 '--jq', '.content'],
                capture_output=True, text=True, timeout=10)
            if result.returncode != 0:
                continue
            content = base64.b64decode(result.stdout.strip()).decode('utf-8')
            for m in re.finditer(r'display_name\s*=\s*\"([^\"]+)\"', content):
                dn = m.group(1)
                mapping[dn] = {
                    'repo': repo,
                    'module': squad,
                    'file': f'tf/squads/{squad}/{fname}',
                    'type': 'squad_definition'
                }
        except Exception:
            continue

# Step 4: Scan root-level .tf files for module invocations
try:
    result = subprocess.run(
        ['gh', 'api', f'repos/{repo}/contents/tf', '--jq', '.[].name'],
        capture_output=True, text=True, timeout=10)
    root_files = [f for f in result.stdout.strip().split('\n')
                  if f.endswith('.tf')]
except Exception:
    root_files = []

for fname in root_files:
    try:
        result = subprocess.run(
            ['gh', 'api', f'repos/{repo}/contents/tf/{fname}',
             '--jq', '.content'],
            capture_output=True, text=True, timeout=10)
        if result.returncode != 0:
            continue
        content = base64.b64decode(result.stdout.strip()).decode('utf-8')
        # Extract module invocations: module \"name\" { source = \"./modules/X\"
        for m in re.finditer(
            r'module\s+\"([^\"]+)\"\s*\{[^}]*source\s*=\s*\"([^\"]+)\"',
            content, re.DOTALL):
            mod_name = m.group(1)
            mod_source = m.group(2)
            # Store invocation location for modules we already know about
            mod_key = mod_source.split('/')[-1] if '/' in mod_source else mod_source
            for dn, info in mapping.items():
                if info['module'] == mod_key:
                    info['invocation_file'] = f'tf/{fname}'
    except Exception:
        continue

with open(f'{iac_dir}/iac-modules.json', 'w') as f:
    json.dump(mapping, f, indent=2)
with open(f'{iac_dir}/iac-status.json', 'w') as f:
    json.dump({'status': 'ok' if mapping else 'empty',
               'count': len(mapping)}, f)
print(f'Discovered {len(mapping)} display_name → file_path mappings')
"
```

**On-disk artifacts (always written):**
- `iac-discovery/iac-modules.json` — `{display_name: {repo, module, file, type, invocation_file?}}`
- `iac-discovery/iac-status.json` — `{"status": "ok|empty|unavailable", "count": N}`

**Display-name matching rules:**
- The PromQL `display_name` in TF often contains Terraform variable interpolations (e.g., `"[${var.cluster}] Mobile API HTTP 5xx Error Rate (critical)"`). Strip `${var.XXX}` to a regex-compatible `.*` and match against the policy's `display_name` from `policies.json`.
- If a policy `display_name` matches multiple TF display_names (e.g., parameterized modules instantiated per-cluster), prefer the module definition file over squad files.

**Fallback: repo-level search link.** When a policy `display_name` does not match any discovered TF file, generate a fallback link to the repo tree for manual search: `https://github.com/{org}/{repo}/tree/main/tf`. This is always more useful than a GitHub code search for a policy ID.

#### Optional: Ownership Resolution via CODEOWNERS

After IaC module discovery, resolve suggested owners for policies that lack a `squad` label by cross-referencing IaC file paths against CODEOWNERS.

**Two-layer resolution:**

1. **CODEOWNERS match (mechanical):** For each IaC file path in `iac-modules.json`, find the matching CODEOWNERS rule (last match wins, per git spec). This resolves policies whose module definition or invocation file has a specific CODEOWNERS entry. Typically resolves high-volume infrastructure alerts (pod restarts, message broker, MySQL, ingress, WAF).

2. **Squad-directory inference (structural):** For policies defined or invoked from `tf/squads/{squad}/*.tf`, the squad directory itself implies ownership — even if the CODEOWNERS rule for that path is generic. A module invocation in `tf/squads/cosmos/recommendation_alerts.tf` means cosmos owns that alert regardless of whether CODEOWNERS maps `tf/squads/cosmos/*` to `@org/cosmos` or a generic owner. The squad directory name is the authoritative signal.

**Flow:**

```bash
# Fetch CODEOWNERS (single gh api call)
gh api "repos/$REPO/contents/CODEOWNERS" --jq '.content' 2>/dev/null \
  | base64 -d > "$IAC_DIR/CODEOWNERS" 2>/dev/null
```

Then in Python:

```python
# Parse CODEOWNERS rules
codeowner_rules = []  # [(glob_pattern, [owners])]
for line in open(f'{iac_dir}/CODEOWNERS'):
    line = line.strip()
    if not line or line.startswith('#'):
        continue
    parts = line.split()
    codeowner_rules.append((parts[0], [p for p in parts[1:] if p.startswith('@')]))

def resolve_owner(filepath):
    """Last matching CODEOWNERS rule wins (git spec)."""
    matched = None
    for pattern, owners in codeowner_rules:
        # Convert CODEOWNERS glob to simple match
        if fnmatch.fnmatch(filepath, pattern) or filepath.startswith(pattern.rstrip('*')):
            matched = owners
    return matched

# For each entry in iac-modules.json:
# 1. Check if file is in tf/squads/{squad}/ → squad name is the owner
# 2. Else resolve via CODEOWNERS match on the file path
# 3. Store as 'suggested_owner' in the mapping
```

**On-disk artifacts (appended to existing):**
- `iac-discovery/iac-modules.json` — each entry gains `suggested_owner` field (team handle or null)
- `iac-discovery/codeowners-status.json` — `{"status": "ok|unavailable", "rules": N}`

**Owner resolution priority for report "Target Owner" field:**
1. Policy's existing `squad` label (already set in TF via `user_labels`) — highest priority, use as-is
2. Squad-directory inference from IaC file path (e.g., `tf/squads/cosmos/` → `@org/cosmos`)
3. CODEOWNERS match on IaC file path (e.g., `tf/modules/message_broker_alerts/` → `@org/devops-codeowners`)
4. No match → `⚠ assign`

**Report integration:**
- When `suggested_owner` is available and the policy has no squad label, use `suggested_owner` in the "Target Owner" field with a note: `{suggested_owner} (from CODEOWNERS)` or `{squad_name} (from squad directory)`
- In the Systemic Issues > Ownership/Routing Debt section, report resolution coverage: `"{N}/{M} ownerless clusters resolved via CODEOWNERS/squad-directory inference"`

**Stage 1 report line additions:**
- `"{N} IaC module mappings discovered"` when status is `ok`, `"IaC discovery: {reason}"` otherwise
- `"{N} CODEOWNERS rules loaded, {M} ownerless clusters resolved"` when CODEOWNERS available

### Stage 2: Compute Cluster Stats

Run compute-clusters.py. This produces a structured output with three sections:

**Per-cluster fields (`clusters` array):**
- `raw_incidents`, `deduped_episodes`, `dedupe_window_sec`
- `distinct_resources`, `median_duration_sec`, `median_retrigger_sec`
- `total_open_hours` — sum of actual incident durations (openTime to closeTime) in hours. **MANDATORY source for all open-incident-hours figures in the report.** Do NOT compute open-hours ad-hoc from `raw * median_duration` — that formula is inaccurate in both directions (overstates retrigger clusters, understates chronic long-tail clusters). If this field is missing or 0 for a cluster, the installed script version may predate this feature — re-run with the local source script.
- `tod_pattern`, `pattern` (flapping/chronic/recurring/burst/isolated)
- `noise_score`, `noise_reasons`, `label_inconsistency`
- `threshold_value`, `comparison`, `condition_filter`, `condition_query`, `condition_type`, `condition_match`
- `service_key` (normalized service identity from condition labels, or null), `signal_family` (error_rate/latency/availability/other)

**Inventory-level fields:**
- `metric_types_in_inventory` — deduplicated set of all metric types across all policy conditions
- `unlabeled_ranking` — top 20 enabled policies without squad/team/owner label, ranked by total raw incidents

### Stage 3: Classify and Prescribe

Read the cluster stats JSON. For each cluster, apply the prescriptive reasoning templates below to assign a verdict and specific recommended action.

**Threshold-aware prescriptions:** When the cluster data includes `threshold_value` and `comparison`, state the current config explicitly in the per-item output: *"Current: {comparison} {threshold_value}, eval window {eval_window_sec}s, auto_close {auto_close_sec}s"*. Use the actual value in recommendations: *"Raise threshold from 0 to >1000"* not *"Raise threshold to >1000"*. When `condition_match` is `"ambiguous"`, note all threshold values and flag which one the recommendation applies to. When `condition_filter` or `condition_query` is available, include a truncated excerpt (≤60 chars) in the per-item output for reviewer auditability.

### Stage 3b: Targeted Metric Validation

For the **top 5 clusters by raw incidents** where the evidence basis would be "heuristic" (not "structural" — structural flaws like auto_close NOT SET don't need metric validation):

**Step 1 — Map PromQL metric names to Cloud Monitoring equivalents.**

Most PromQL conditions use metrics that ARE queryable via Cloud Monitoring under a mapped name:

| PromQL pattern | Cloud Monitoring equivalent | Example |
|---|---|---|
| `kubernetes_io:METRIC_PATH` | `kubernetes.io/METRIC_PATH` | `kubernetes_io:container_restart_count` → `kubernetes.io/container/restart_count` |
| `SERVICE_COM:METRIC` | `SERVICE.com/METRIC` | `dbinsights_googleapis_com:perquery_latencies` → `dbinsights.googleapis.com/perquery/latencies` |
| `METRIC_NAME` (custom) | `prometheus.googleapis.com/METRIC_NAME/SUFFIX` | `jvm_threads_live_threads` → `prometheus.googleapis.com/jvm_threads_live_threads/gauge` |
| `METRIC_NAME_count` (histogram _count) | `prometheus.googleapis.com/METRIC_NAME/histogram` | `http_server_duration_milliseconds_count` → `prometheus.googleapis.com/http_server_duration_milliseconds/histogram` (use `distributionValue.count` for request counts) |

For custom Prometheus metrics, the suffix depends on the metric type: try `/gauge` first, then `/counter`, `/summary`, `/histogram` if no data returns. **Important:** When PromQL references `METRIC_count` (the `_count` component of a histogram), the Cloud Monitoring equivalent is the parent histogram type (`/histogram`), NOT a separate `_count` metric. The request count is in the `distributionValue.count` field of the distribution result. If none return data, list metric descriptors with `filter=metric.type = starts_with("prometheus.googleapis.com/METRIC_PREFIX")` to find the actual suffix.

**Label discovery from PromQL:** Cloud Monitoring labels don't always match what you'd guess. Read the PromQL query from `policies.json` to find the actual label names used. For example, `http_server_requests_seconds_count{container="hcs-gb"}` tells you the Cloud Monitoring filter needs `metric.labels.container = "hcs-gb"` — not `metric.labels.application` or `metric.labels.service`. Always check the PromQL before constructing filters. PromQL labels on the metric selector map to `metric.labels.X` in Cloud Monitoring.

**Step 2 — Extract threshold from PromQL query text.**

When `threshold_value` is null (PromQL conditions), extract from the query string. PromQL thresholds appear as a comparison operator at the end or after a closing parenthesis: `... > 1000`, `... < 0.01`, `... == 0`. Match with: `[><=!]+\s*([\d.]+)\s*$` on the query text.

**Mandatory Query Parameters by Metric Class:**

Every metric validation query MUST specify ALL THREE aggregation parameters. Omitting `crossSeriesReducer` when multiple series exist produces sparse, unrepresentative samples.

| Metric class | perSeriesAligner | alignmentPeriod | crossSeriesReducer | Expected points (14d) |
|---|---|---|---|---|
| Gauge (JVM threads, memory util, disk util) | ALIGN_MAX | 3600s | REDUCE_MAX | ~336 |
| Counter/delta (request count, restart count) | ALIGN_DELTA | 3600s | REDUCE_SUM | ~336 |
| Distribution (DB Insights latency) | ALIGN_DELTA | 86400s | — (per-hash) | ~14 |
| Error rate (ratio of two counters) | ALIGN_DELTA + REDUCE_SUM | 3600s | REDUCE_SUM (per query) | ~336 each |

When in doubt, use REDUCE_MAX for gauges and REDUCE_SUM for counters. Never omit the reducer for metrics that may have multiple series (per-pod, per-container, per-node).

**Step 3 — Query the metric.**

Use MCP `list_time_series` (Tier 1) or REST API via temp-file pattern (Tier 2). Select aligner and reducer from the Mandatory Query Parameters table above based on the metric class.

**Tier 2 query pattern** (uses session temp file per Behavioral Constraint 2):

```bash
WORK_DIR="${WORK_DIR:-$(mktemp -d /tmp/ah-XXXXXX)}"
FILTER_FILE=$(mktemp "$WORK_DIR/ts-filter-XXXXXX.txt")
cat > "$FILTER_FILE" << 'FILTER'
metric.type = "prometheus.googleapis.com/jvm_threads_live_threads/gauge"
AND metric.labels.container = "recommendation-service"
AND resource.labels.cluster = "example-cluster-prod"
FILTER

TOKEN=$(gcloud auth print-access-token)
FILTER_ENC=$(python3 -c "import urllib.parse, sys; print(urllib.parse.quote(open(sys.argv[1]).read().strip().replace('\n', ' ')))" "$FILTER_FILE")

# Note: REDUCE_MAX for gauges; use REDUCE_SUM for counters per the table above
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://monitoring.googleapis.com/v3/projects/$PROJECT/timeSeries?filter=$FILTER_ENC&interval.startTime=${START_TIME}&interval.endTime=${END_TIME}&aggregation.alignmentPeriod=3600s&aggregation.perSeriesAligner=ALIGN_MAX&aggregation.crossSeriesReducer=REDUCE_MAX&pageSize=100000" \
  -o "$WORK_DIR/ts-result.json" ; rm -f "$FILTER_FILE"
```

Write results to `$WORK_DIR`, not to conversation context. Parse with `python3` one-liner to extract p50/p95.

**Result Quality Gate — MANDATORY before using any metric result:**

After every query, validate the result before computing percentiles:

1. **Point count check:** For a 14-day window with 1h alignment, expect ~336 points per reduced series. If fewer than 50 points are returned, flag the result as `insufficient sample` and note the actual count. Do NOT use sparse results to derive threshold recommendations — they produce unreliable percentiles. For distribution metrics with daily alignment (~14 expected points), use a minimum threshold of 7 points instead of 50.
2. **Series count check:** After REDUCE_MAX/REDUCE_SUM, expect exactly 1 series. If >1 series is returned, the reducer was not applied — re-query with the correct reducer.
3. **Pagination check:** If the response contains `nextPageToken`, the result is truncated. Re-query with higher `pageSize` or paginate. Do NOT compute percentiles from truncated data.
4. **Evidence block logging:** Every metric validation result MUST log in the Evidence Ledger: metric type, filter, aligner, reducer, alignment period, series count, point count, and numeric results. This makes cross-report comparisons auditable.

If a query fails the quality gate, mark the evidence basis as `heuristic (insufficient sample: N points)` instead of `measured`. Do not promote to Do Now based on insufficient samples.

**Step 4 — Compare against threshold.**

- If p95 < threshold: the alert rarely fires legitimately. Recommend lowering threshold to p95 + 20% headroom. State: *"p95={X}, threshold={Y}, headroom={Z}%"*.
- If p50 > 0.8 × threshold: baseline crowding. State: *"p50={X} is {Y}% of threshold={Z} — baseline crowding"*.
- Record the query used and numeric result in the cluster's `Evidence basis` field. Mark as `measured`.

**Step 5 — Handle failures.**

If the metric type cannot be mapped, the query returns no data, the suffix is wrong, or the result fails the quality gate: note the reason and keep `heuristic` basis. A no-data result is itself a finding — the metric may be misconfigured or the resource labels may not match. An insufficient-sample result (< 50 points) means the query parameters need adjustment (missing reducer, low pageSize, or metric sparsity) — do not treat it as measured evidence.

**Scope:** Query all clusters, not just heuristic-basis. Structural items also benefit from measurement (e.g., confirming queue baselines are zero proves the `> 0` threshold fires on transients). If API errors or rate limits occur, prioritize by raw incident count.

**Multi-query patterns** — these are NOT limitations, they are standard approaches. Use them:

| Metric kind | Approach | Example |
|---|---|---|
| **CUMULATIVE DISTRIBUTION** (e.g., perquery latencies) | `ALIGN_DELTA` extracts the delta distribution per period. Read `distributionValue.mean` for average latency and `distributionValue.count` for query volume. | `dbinsights.googleapis.com/perquery/latencies` + ALIGN_DELTA daily → mean latency per day |
| **Error rate ratios** (5xx / total) | Two queries: (1) filter by status label for errors, (2) unfiltered for total. Divide in Python. Write both results to `$WORK_DIR`. | Query with `metric.labels.status=starts_with("5")` for errors, without for total. Compute `error_count / total_count`. |
| **High-cardinality counters** (e.g., 1,000+ containers) | Use `aggregation.crossSeriesReducer=REDUCE_SUM` to aggregate across all series. Gives total restart rate without enumerating each container. | `kubernetes.io/container/restart_count` + ALIGN_DELTA 1h + REDUCE_SUM → total restarts/hour across all containers |
| **Histogram-based error rates** (e.g., `rate(metric_count{status=~"5.."}[5m])`) | When PromQL uses `_count` from a histogram metric (e.g., `http_server_duration_milliseconds_count`), the Cloud Monitoring equivalent is the `/histogram` type — NOT a separate `_count` metric. Query `prometheus.googleapis.com/METRIC_NAME/histogram` with `ALIGN_DELTA`. The `distributionValue.count` field gives the request count per period. For error-rate ratios: query with status label filter for 5xx (count = error volume), then without filter (count = total volume). Compute ratio from the two `count` sums. | `prometheus.googleapis.com/http_server_duration_milliseconds/histogram` + ALIGN_DELTA 1h → `distributionValue.count` per period. Two queries: with `http_status_code=~"5.."` for errors, without for total. |
| **Custom PromQL with no Cloud Monitoring equivalent** | Rare — most Prometheus metrics map via `prometheus.googleapis.com/METRIC/SUFFIX`. Try `/gauge`, `/counter`, `/summary`, `/histogram` suffixes. If none return data, list metric descriptors with `starts_with` filter. | `metricDescriptors?filter=metric.type = starts_with("prometheus.googleapis.com/METRIC_PREFIX")` |

**Tier 2 two-query ratio pattern** (error rate example):

```bash
WORK_DIR="${WORK_DIR:-$(mktemp -d /tmp/ah-XXXXXX)}"

# Query 1: error count (5xx)
FILTER_ERR=$(mktemp "$WORK_DIR/ts-err-XXXXXX.txt")
cat > "$FILTER_ERR" << 'FILTER'
metric.type = "prometheus.googleapis.com/http_server_requests_seconds_count/summary"
AND metric.labels.status = starts_with("5")
AND metric.labels.application = "SERVICE_NAME"
FILTER
# ... curl with crossSeriesReducer=REDUCE_SUM&pageSize=100000 to $WORK_DIR/ts-err-result.json

# Query 2: total count
FILTER_TOTAL=$(mktemp "$WORK_DIR/ts-total-XXXXXX.txt")
cat > "$FILTER_TOTAL" << 'FILTER'
metric.type = "prometheus.googleapis.com/http_server_requests_seconds_count/summary"
AND metric.labels.application = "SERVICE_NAME"
FILTER
# ... curl with crossSeriesReducer=REDUCE_SUM&pageSize=100000 to $WORK_DIR/ts-total-result.json

# Compute ratio
python3 -c "
import json
err = sum(float(pt['value']['doubleValue']) for s in json.load(open('$WORK_DIR/ts-err-result.json')).get('timeSeries',[]) for pt in s['points'] if 'doubleValue' in pt.get('value',{}))
total = sum(float(pt['value']['doubleValue']) for s in json.load(open('$WORK_DIR/ts-total-result.json')).get('timeSeries',[]) for pt in s['points'] if 'doubleValue' in pt.get('value',{}))
print(f'error_rate={err/total*100:.2f}% ({err:.0f}/{total:.0f})') if total > 0 else print('no data')
"
rm -f "$FILTER_ERR" "$FILTER_TOTAL"
```

If a multi-query approach is available but not executed (e.g., auth expired), state the approach and mark as "not attempted — auth expired" rather than claiming an API limitation.

### Stage 4: Coverage Gap Check

Read `metric_types_in_inventory` from the compute-clusters output. Compare against the Coverage Gap Checklist below. For each gap:
- If the metric type exists in the inventory: check scope — is it applied to all relevant clusters/services, or only a subset? If partial, report "extend scope" with the specific uncovered projects.
- If the metric type does not exist: report "add new".
- Cross-reference each gap with existing noisy clusters that the gap metric would detect upstream (e.g., probe failure coverage → would provide early signal for pod restart clusters).

#### Routing Validation

Check for routing and ownership gaps using policy-level and cluster-level data from `$WORK_DIR`.

**Zero-channel policies (policy-level):**
Scan `policies.json` for entries where `notificationChannels` is empty AND `enabled` is true. These fire but notify nobody — invisible failures.
- Policies with `raw_incidents > 10` (cross-reference against `clusters.json`): promote to **Investigate** with action: *"Silent alert — fires but has no notification channels. Add a notification channel or disable."*
- Policies with low/zero incidents: surface in **Systemic Issues > Dead/Orphaned Config** — *"Zero-channel policy: {displayName} — enabled but no notification path."*

**Unlabeled high-noise policies (policy-level):**
The existing `unlabeled_ranking` table stays in Systemic Issues. Additionally, for top entries with `raw_incidents > 10`:
- Promote to **Investigate** with ownership language: *"No squad/team/owner label — ownership is implicit and unauditable. {N} incidents in {days}d have no traceable owner."*
- Do not reference org-specific routing fallback channels. The skill is generic.
- Leave suggested owner as "⚠ assign" — the skill does not infer squad ownership from service names.

**Label inconsistency promotion (cluster-level):**
For clusters where `label_inconsistency` is true AND `raw_incidents > 5`:
- Promote to **Investigate** as a cluster-level finding: *"Label mismatch — staging/test label on a production resource. {N} incidents may be misrouted or ignored."*
- Do NOT merge into the policy-level unlabeled table — entity types stay separate.

#### SLO Coverage Cross-Reference

Only run this section when `$WORK_DIR/slo-source-status.json` has `"status": "ok"`. Read `$WORK_DIR/slo-services.json` (normalized service names) and cross-reference against cluster data.

**Service grouping:** Group clusters by `service_key` from `clusters.json`. Skip clusters where `service_key` is `null`. Only consider clusters with `signal_family` in (`error_rate`, `latency`, `availability`) — exclude `other`.

**SLO migration candidates:**
For each service_key with `total_raw_incidents > 20` across user-facing clusters AND NOT in `slo-services.json`:
- Surface in **Coverage Gaps** table: *"SLO review candidate — {service_key} has {N} noisy user-facing threshold alerts ({families}), no SLO definition."*

**SLO review candidates (redundancy check):**
For each service_key that IS in `slo-services.json` AND has noisy user-facing threshold alerts:
- Surface as **Needs Decision** item: *"Service {service_key} has an SLO definition and also {N} noisy user-facing threshold alerts; review for redundancy or intentional overlap."*
- Do not claim same-signal overlap — `slo-services.json` carries service names only, not signal coverage metadata.

**Matching rules:** Both `service_key` and SLO service names are normalized (lowercase, env suffixes stripped, separators unified). Only exact-match after normalization. Unmatched service_keys are excluded, not fuzzy-guessed.

### Stage 5: Produce Report

Write the final report as markdown using the Report Skeleton template below. Group findings by action class (Do Now / Investigate / Needs Decision), not by confidence band. Apply the Do Now Gate to determine which items qualify for the Do Now section. Items that fail the gate drop to Investigate regardless of confidence level.

#### IaC Location Resolution via Module Discovery

During Do Now gate evaluation, resolve IaC Location for each finding using the `iac-modules.json` mapping built in Stage 1.

**Resolution rules (in priority order):**

1. **Confirmed:** The policy's `display_name` matches a `display_name` extracted from a TF file in `iac-modules.json` (accounting for `${var.XXX}` interpolation). The file path is known and verified from the repo tree. Use direct link.
2. **Likely:** The policy's metric type, PromQL fragment, or resource type matches a known TF module name (e.g., policy using `artemis_consumer_count` → module `message_broker_alerts`), but `display_name` didn't match exactly (e.g., due to complex interpolation). Include the candidate file path.
3. **Search Required:** No match in `iac-modules.json`, but `{iac_repos}` were provided. Include a link to the repo tree (`https://github.com/{org}/{repo}/tree/main/tf`) for manual search, plus the policy display_name as a search hint.
4. **Unknown:** No `{iac_repos}` provided or IaC discovery failed entirely.

**Link construction:**
- **Confirmed/Likely:** Direct link to the file: `https://github.com/{org}/{repo}/blob/main/{file_path}`
- **Search Required:** Link to the TF directory: `https://github.com/{org}/{repo}/tree/main/tf`
- If an `invocation_file` exists (module is instantiated from a separate file with threshold/label variables), include it as a second link: the module definition for the PromQL template, and the invocation file for the configurable values (thresholds, labels, duration).

**Do not use `gh search code`** for IaC resolution. Policy IDs are GCP-generated at `terraform apply` time and never appear in `.tf` source. GitHub code search for policy IDs always returns zero results.

## Prescriptive Reasoning by Pattern

### Flapping (raw/episode > 3, raw > 10)

The alert fires and resolves repeatedly on the same resource. Root cause: threshold too close to baseline, or evaluation window too short.

Compare the configured threshold to the median retrigger interval and median duration:
- If median_duration < duration setting: raise duration to 2x median_duration
- If threshold is below observed baseline (from metric validation): raise threshold above p50 of the observed metric. State current vs recommended value.
- If auto_close is NOT SET and median_duration > 1h: set auto_close to 2x median_duration (capped at 86400s / 24h)
- If notification_channels > 0 and noise_score >= 5: recommend demoting to dashboard-only
- If raw/episode ratio > 10: the alert has a structural flap design — recommend auto_close or redesign

### Chronic (episodes >= 5, median_duration > 1h)

The alert fires and stays open for extended periods. The underlying issue is real and persistent.

This is "Fix the underlying issue". The alert is working correctly — the service has a problem:
- State specific investigation pointers based on metric_type and resource_type
- If auto_close is NOT SET: recommend setting it to match the evaluation window to prevent alert accumulation
- Do not recommend raising the threshold — the alert is detecting a real condition
- If distinct_resources is high (> 20): flag as systemic/cluster-wide, not single-service

### Recurring (episodes >= 5, median_duration < 1h)

The alert fires regularly but resolves within an hour. Could be deploy-time transients, batch jobs, or genuine intermittent issues.

- If tod_pattern shows > 50% concentration at a specific hour: infer scheduled job or deployment window. Recommend time-of-day mute for that window or extend duration to ride through the transient. Flag as low confidence if < 80% concentration.
- If

…(truncated)