Alert Hygiene

Alert Hygiene Analysis

Use when facing alert noise, flapping alerts, alert fatigue, threshold audits, or SLO-alert redesign questions. Analyzes alert policies and incident history for a single GCP monitoring project. Produces an action-class prescriptive report: Do Now items (PR-ready config changes with strict gating), Investigate items (bounded discovery with two-stage DoD), and Needs Decision items (strategy/policy choices with named owners). Runs straight through from data pull to final report.

When NOT to use:

• Active incident investigation — use incident-analysis instead (tiered log investigation with mitigation playbooks)
• SLO design from scratch — this skill identifies SLO-redesign candidates, it does not design SLOs or generate burn-rate PromQL
• Multi-project analysis — v1 is scoped to one monitoring project per invocation

Behavioral Constraints

1. Scope Restriction

All queries are scoped to one monitoring project per invocation. The user must provide the monitoring project ID before analysis begins. If not provided, ask once.

2. Temp-File Execution Pattern

All bulk data operations write to session-scoped temp files via mktemp. This keeps large JSON payloads out of conversation context and makes the shell path debuggable:

WORK_DIR=$(mktemp -d /tmp/ah-XXXXXX)

All scripts read from and write to $WORK_DIR. Clean up at end of session.

3. No Checkpoint Gates

The analysis runs straight through from data pull to final report. The frequency table appears in the report appendix, not as a separate pause point. Do not ask the user for input mid-analysis.

4. Confidence Inline Flags

When an inference has less than high confidence (e.g., inferring a nightly batch from time-of-day concentration), state the inference and flag it inline: "Inferred nightly batch from 100% h02 concentration — verify before applying mute window." Do not gate the report on user confirmation.

Tier Detection

Run the shared observability preflight before data access:

bash "$(dirname "$0")/../scripts/obs-preflight.sh"

If gcloud is unauthenticated, run gcloud auth login before proceeding. If gcloud is missing, fall to Tier 3.

Tier 1 — REST API via scripts (default for inventory and bulk)

Default path for all data pull. Uses gcloud auth print-access-token for auth, curl for requests, Python scripts for pagination and extraction. MCP truncates at ~100K characters and cannot handle monitoring projects with >50 policies — REST has no such limits.

WORK_DIR=$(mktemp -d /tmp/ah-XXXXXX)
SCRIPTS_DIR="$(dirname "$0")/../skills/alert-hygiene/scripts"

python3 "$SCRIPTS_DIR/pull-policies.py" \
    --project "$MONITORING_PROJECT" \
    --output "$WORK_DIR/policies.json"

python3 "$SCRIPTS_DIR/pull-incidents.py" \
    --project "$MONITORING_PROJECT" \
    --days 14 \
    --output "$WORK_DIR/alerts.json"

python3 "$SCRIPTS_DIR/compute-clusters.py" \
    --policies "$WORK_DIR/policies.json" \
    --alerts "$WORK_DIR/alerts.json" \
    --output "$WORK_DIR/clusters.json"

Tier 2 — MCP (targeted enrichment only)

If list_time_series or list_alert_policies MCP tools are available, use them only for targeted enrichment after cluster stats are computed:

• Single policy detail lookups (reading full PromQL query text)
• Time-series queries for metric validation on nominated clusters (< 50 results expected)

Never use MCP for bulk inventory pulls or incident history.

Tier 3 — Guidance only

If neither gcloud nor MCP tools are available, provide manual Cloud Console instructions.

Analysis Flow

Stage 0: Validate Data Access

Before any analysis, discover and validate the monitoring project:

1. If the user provides a project ID, verify it is a monitoring project (has alert policies), not just a resource project. Run: python3 pull-policies.py --project PROJECT --output /tmp/ah-check.json.
2. If zero policies returned: The project is likely a resource project scoped under a different monitoring project. Provide actionable guidance: "Project {X} returned 0 alert policies. This is likely a resource project, not the monitoring project. Try: (a) project ending in -monitoring (e.g., {org}-monitoring), (b) check Cloud Console > Monitoring > Settings > Metrics Scope to find which project hosts the alert policies, (c) run gcloud alpha monitoring policies list --project={candidate} to test candidates." Do not just ask the user — give them the diagnostic path.
3. If the user does not provide a project, check if recent incident context or prior conversation names it. If not, ask once with the guidance above.
4. Verify incidents return non-empty: run pull-incidents.py with --days 1 as a quick check. If zero incidents but policies exist, the window may be too narrow or incidents are in a different project — warn with specifics.
5. If both return data: "Monitoring project: {project}, {N} policies found, incidents available. Proceeding with {days}-day analysis."
6. GitHub org and IaC repos for links: Check if recent conversation or CLAUDE.md names a GitHub org and IaC repos. If not, ask once: "Which GitHub org hosts your monitoring IaC, and which repo(s)? (e.g., example-org, repos monitoring and k8s). Used for IaC links in the report — skip if not applicable." Store as {github_org} and {iac_repos} (list). If the user skips, omit IaC links.

Fail early — do not proceed to Stage 1 with an empty or wrong project.

Stage 1: Pull Data

Run pull-policies.py and pull-incidents.py with the validated project. Verify non-empty results. Report counts:

• Total policies (enabled/disabled)
• Total incidents in window
• Policies by squad label

Optional: SLO Config Enrichment

After pulling policies and incidents, attempt to fetch SLO service definitions from GitHub. This is optional enrichment — the core analysis runs without it.

Preconditions: {github_org} was provided in Stage 0 AND gh CLI is available AND authenticated (gh auth status succeeds). If any precondition fails, write fallback artifacts and skip.

Flow:

REPO="${github_org}/monitoring"

# Short-circuit if preconditions not met
if [ -z "${github_org:-}" ]; then
  echo '{"status":"unavailable","reason":"github_org_not_provided","count":0}' \
    > "$WORK_DIR/slo-source-status.json"
  echo '[]' > "$WORK_DIR/slo-services.json"
elif ! command -v gh &>/dev/null || ! gh auth status &>/dev/null 2>&1; then
  echo '{"status":"unavailable","reason":"gh_not_available","count":0}' \
    > "$WORK_DIR/slo-source-status.json"
  echo '[]' > "$WORK_DIR/slo-services.json"
else
  # Step 1: fetch (checked)
  if ! gh api "repos/$REPO/contents/tf/slo-config.yaml" \
       --jq '.content' > "$WORK_DIR/slo-raw-b64.txt" 2>"$WORK_DIR/slo-fetch-err.txt"; then
    echo '{"status":"unavailable","reason":"fetch_failed","count":0}' \
      > "$WORK_DIR/slo-source-status.json"
    echo '[]' > "$WORK_DIR/slo-services.json"
  # Step 2: decode (checked)
  elif ! base64 -d < "$WORK_DIR/slo-raw-b64.txt" > "$WORK_DIR/slo-config.yaml" 2>/dev/null; then
    echo '{"status":"unavailable","reason":"decode_failed","count":0}' \
      > "$WORK_DIR/slo-source-status.json"
    echo '[]' > "$WORK_DIR/slo-services.json"
  # Step 3: extract service names (checked — Ruby stdlib YAML, no PyYAML)
  elif ! ruby -ryaml -rjson -e '
    cfg = YAML.safe_load(File.read(ARGV[0])) || {}
    services = (cfg["services"] || []).map { |s| s["name"] }.compact
      .map { |n| n.downcase.gsub(/[_.]/, "-").gsub(/-(prod|staging|pta)$/, "") }
    File.write(ARGV[1], JSON.generate(services))
    File.write(ARGV[2], JSON.generate({
      status: services.empty? ? "empty" : "ok",
      count: services.length
    }))
  ' "$WORK_DIR/slo-config.yaml" "$WORK_DIR/slo-services.json" \
    "$WORK_DIR/slo-source-status.json" 2>"$WORK_DIR/slo-parse-err.txt"; then
    echo '{"status":"unavailable","reason":"parse_failed","count":0}' \
      > "$WORK_DIR/slo-source-status.json"
    echo '[]' > "$WORK_DIR/slo-services.json"
  fi
  rm -f "$WORK_DIR/slo-raw-b64.txt"
fi

On-disk artifacts (always written, every branch):

• slo-services.json — normalized service name list or []
• slo-source-status.json — {"status": "ok|empty|unavailable", "reason": "...", "count": N}

Names in slo-services.json are normalized with the same rules as service_key in compute-clusters.py: lowercase, strip environment suffixes (-prod, -staging, -pta), replace separators (_, .) with -.

Stage 1 report line addition: "{M} services with SLO definitions" when status is ok, "SLO config: {reason}" otherwise.

Optional: IaC Module Discovery

After SLO enrichment, build a display_name → file_path mapping by walking the IaC repo tree via gh api. This replaces the previous gh search code approach, which returned zero results because GCP-generated policy IDs never appear in Terraform source.

Preconditions: {github_org} and {iac_repos} were provided in Stage 0 AND gh is available AND authenticated. If any fail, write fallback and skip.

Why repo tree walk instead of gh search code:

• Policy IDs are assigned at terraform apply time — they do not exist in .tf files, so searching for them always returns 0 results
• gh search code has a 10 req/min rate limit and frequently hits 403 on private orgs
• gh api (used here) has a 5,000 req/hour limit and works reliably on private repos
• The results are deterministic: if a display_name matches, the file path is confirmed

Flow:

For each repo in {iac_repos} that contains a tf/ directory:

REPO="${github_org}/${iac_repo}"
IAC_DIR="$WORK_DIR/iac-discovery"
mkdir -p "$IAC_DIR"

# Step 1: List module directories
gh api "repos/$REPO/contents/tf/modules" --jq '.[].name' \
  > "$IAC_DIR/module-names.txt" 2>/dev/null

# Step 2: For each module, fetch the main TF file and extract display_name values
python3 -c "
import subprocess, json, re, os
repo = '$REPO'
iac_dir = '$IAC_DIR'
modules = open(f'{iac_dir}/module-names.txt').read().strip().split('\n')
mapping = {}
for mod in modules:
    # Try common TF file names in the module
    for fname in ['main.tf', 'google_monitoring_alert_policy.tf', f'{mod}.tf']:
        try:
            result = subprocess.run(
                ['gh', 'api', f'repos/{repo}/contents/tf/modules/{mod}/{fname}',
                 '--jq', '.content'],
                capture_output=True, text=True, timeout=10)
            if result.returncode != 0:
                continue
            import base64
            content = base64.b64decode(result.stdout.strip()).decode('utf-8')
            # Extract display_name values from TF
            for m in re.finditer(r'display_name\s*=\s*\"([^\"]+)\"', content):
                dn = m.group(1)
                # Skip condition display names (inside conditions {} blocks)
                # Only keep top-level policy display_name
                mapping[dn] = {
                    'repo': repo,
                    'module': mod,
                    'file': f'tf/modules/{mod}/{fname}',
                    'type': 'module_definition'
                }
            break  # Found file, stop trying alternatives
        except Exception:
            continue

# Step 3: Also scan squad directories for module invocations
try:
    result = subprocess.run(
        ['gh', 'api', f'repos/{repo}/contents/tf/squads', '--jq', '.[].name'],
        capture_output=True, text=True, timeout=10)
    squads = result.stdout.strip().split('\n') if result.returncode == 0 else []
except Exception:
    squads = []

for squad in squads:
    try:
        result = subprocess.run(
            ['gh', 'api', f'repos/{repo}/contents/tf/squads/{squad}',
             '--jq', '.[].name'],
            capture_output=True, text=True, timeout=10)
        files = [f for f in result.stdout.strip().split('\n')
                 if f.endswith('.tf')]
    except Exception:
        continue
    for fname in files:
        try:
            result = subprocess.run(
                ['gh', 'api',
                 f'repos/{repo}/contents/tf/squads/{squad}/{fname}',
                 '--jq', '.content'],
                capture_output=True, text=True, timeout=10)
            if result.returncode != 0:
                continue
            content = base64.b64decode(result.stdout.strip()).decode('utf-8')
            for m in re.finditer(r'display_name\s*=\s*\"([^\"]+)\"', content):
                dn = m.group(1)
                mapping[dn] = {
                    'repo': repo,
                    'module': squad,
                    'file': f'tf/squads/{squad}/{fname}',
                    'type': 'squad_definition'
                }
        except Exception:
            continue

# Step 4: Scan root-level .tf files for module invocations
try:
    result = subprocess.run(
        ['gh', 'api', f'repos/{repo}/contents/tf', '--jq', '.[].name'],
        capture_output=True, text=True, timeout=10)
    root_files = [f for f in result.stdout.strip().split('\n')
                  if f.endswith('.tf')]
except Exception:
    root_files = []

for fname in root_files:
    try:
        result = subprocess.run(
            ['gh', 'api', f'repos/{repo}/contents/tf/{fname}',
             '--jq', '.content'],
            capture_output=True, text=True, timeout=10)
        if result.returncode != 0:
            continue
        content = base64.b64decode(result.stdout.strip()).decode('utf-8')
        # Extract module invocations: module \"name\" { source = \"./modules/X\"
        for m in re.finditer(
            r'module\s+\"([^\"]+)\"\s*\{[^}]*source\s*=\s*\"([^\"]+)\"',
            content, re.DOTALL):
            mod_name = m.group(1)
            mod_source = m.group(2)
            # Store invocation location for modules we already know about
            mod_key = mod_source.split('/')[-1] if '/' in mod_source else mod_source
            for dn, info in mapping.items():
                if info['module'] == mod_key:
                    info['invocation_file'] = f'tf/{fname}'
    except Exception:
        continue

with open(f'{iac_dir}/iac-modules.json', 'w') as f:
    json.dump(mapping, f, indent=2)
with open(f'{iac_dir}/iac-status.json', 'w') as f:
    json.dump({'status': 'ok' if mapping else 'empty',
               'count': len(mapping)}, f)
print(f'Discovered {len(mapping)} display_name → file_path mappings')
"

On-disk artifacts (always written):

• iac-discovery/iac-modules.json — {display_name: {repo, module, file, type, invocation_file?}}
• iac-discovery/iac-status.json — {"status": "ok|empty|unavailable", "count": N}

Display-name matching rules:

• The PromQL display_name in TF often contains Terraform variable interpolations (e.g., "[${var.cluster}] Mobile API HTTP 5xx Error Rate (critical)"). Strip ${var.XXX} to a regex-compatible .* and match against the policy's display_name from policies.json.
• If a policy display_name matches multiple TF display_names (e.g., parameterized modules instantiated per-cluster), prefer the module definition file over squad files.

Fallback: repo-level search link. When a policy display_name does not match any discovered TF file, generate a fallback link to the repo tree for manual search: https://github.com/{org}/{repo}/tree/main/tf. This is always more useful than a GitHub code search for a policy ID.

Optional: Ownership Resolution via CODEOWNERS

After IaC module discovery, resolve suggested owners for policies that lack a squad label by cross-referencing IaC file paths against CODEOWNERS.

Two-layer resolution:

1. CODEOWNERS match (mechanical): For each IaC file path in iac-modules.json, find the matching CODEOWNERS rule (last match wins, per git spec). This resolves policies whose module definition or invocation file has a specific CODEOWNERS entry. Typically resolves high-volume infrastructure alerts (pod restarts, message broker, MySQL, ingress, WAF).

2. Squad-directory inference (structural): For policies defined or invoked from tf/squads/{squad}/*.tf, the squad directory itself implies ownership — even if the CODEOWNERS rule for that path is generic. A module invocation in tf/squads/cosmos/recommendation_alerts.tf means cosmos owns that alert regardless of whether CODEOWNERS maps tf/squads/cosmos/* to @org/cosmos or a generic owner. The squad directory name is the authoritative signal.

Flow:

# Fetch CODEOWNERS (single gh api call)
gh api "repos/$REPO/contents/CODEOWNERS" --jq '.content' 2>/dev/null \
  | base64 -d > "$IAC_DIR/CODEOWNERS" 2>/dev/null

Then in Python:

# Parse CODEOWNERS rules
codeowner_rules = []  # [(glob_pattern, [owners])]
for line in open(f'{iac_dir}/CODEOWNERS'):
    line = line.strip()
    if not line or line.startswith('#'):
        continue
    parts = line.split()
    codeowner_rules.append((parts[0], [p for p in parts[1:] if p.startswith('@')]))

def resolve_owner(filepath):
    """Last matching CODEOWNERS rule wins (git spec)."""
    matched = None
    for pattern, owners in codeowner_rules:
        # Convert CODEOWNERS glob to simple match
        if fnmatch.fnmatch(filepath, pattern) or filepath.startswith(pattern.rstrip('*')):
            matched = owners
    return matched

# For each entry in iac-modules.json:
# 1. Check if file is in tf/squads/{squad}/ → squad name is the owner
# 2. Else resolve via CODEOWNERS match on the file path
# 3. Store as 'suggested_owner' in the mapping

On-disk artifacts (appended to existing):

• iac-discovery/iac-modules.json — each entry gains suggested_owner field (team handle or null)
• iac-discovery/codeowners-status.json — {"status": "ok|unavailable", "rules": N}

Owner resolution priority for report "Target Owner" field:

1. Policy's existing squad label (already set in TF via user_labels) — highest priority, use as-is
2. Squad-directory inference from IaC file path (e.g., tf/squads/cosmos/ → @org/cosmos)
3. CODEOWNERS match on IaC file path (e.g., tf/modules/message_broker_alerts/ → @org/devops-codeowners)
4. No match → ⚠ assign

Report integration:

• When suggested_owner is available and the policy has no squad label, use suggested_owner in the "Target Owner" field with a note: {suggested_owner} (from CODEOWNERS) or {squad_name} (from squad directory)
• In the Systemic Issues > Ownership/Routing Debt section, report resolution coverage: "{N}/{M} ownerless clusters resolved via CODEOWNERS/squad-directory inference"

Stage 1 report line additions:

• "{N} IaC module mappings discovered" when status is ok, "IaC discovery: {reason}" otherwise
• "{N} CODEOWNERS rules loaded, {M} ownerless clusters resolved" when CODEOWNERS available

Stage 2: Compute Cluster Stats

Run compute-clusters.py. This produces a structured output with three sections:

Per-cluster fields (clusters array):

• raw_incidents, deduped_episodes, dedupe_window_sec
• distinct_resources, median_duration_sec, median_retrigger_sec
• total_open_hours — sum of actual incident durations (openTime to closeTime) in hours. MANDATORY source for all open-incident-hours figures in the report. Do NOT compute open-hours ad-hoc from raw * median_duration — that formula is inaccurate in both directions (overstates retrigger clusters, understates chronic long-tail clusters). If this field is missing or 0 for a cluster, the installed script version may predate this feature — re-run with the local source script.
• tod_pattern, pattern (flapping/chronic/recurring/burst/isolated)
• noise_score, noise_reasons, label_inconsistency
• threshold_value, comparison, condition_filter, condition_query, condition_type, condition_match
• service_key (normalized service identity from condition labels, or null), signal_family (error_rate/latency/availability/other)

Inventory-level fields:

• metric_types_in_inventory — deduplicated set of all metric types across all policy conditions
• unlabeled_ranking — top 20 enabled policies without squad/team/owner label, ranked by total raw incidents

Stage 3: Classify and Prescribe

Read the cluster stats JSON. For each cluster, apply the prescriptive reasoning templates below to assign a verdict and specific recommended action.

Threshold-aware prescriptions: When the cluster data includes threshold_value and comparison, state the current config explicitly in the per-item output: "Current: {comparison} {threshold_value}, eval window {eval_window_sec}s, auto_close {auto_close_sec}s". Use the actual value in recommendations: "Raise threshold from 0 to >1000" not "Raise threshold to >1000". When condition_match is "ambiguous", note all threshold values and flag which one the recommendation applies to. When condition_filter or condition_query is available, include a truncated excerpt (≤60 chars) in the per-item output for reviewer auditability.

Stage 3b: Targeted Metric Validation

For the top 5 clusters by raw incidents where the evidence basis would be "heuristic" (not "structural" — structural flaws like auto_close NOT SET don't need metric validation):

Step 1 — Map PromQL metric names to Cloud Monitoring equivalents.

Most PromQL conditions use metrics that ARE queryable via Cloud Monitoring under a mapped name:

For custom Prometheus metrics, the suffix depends on the metric type: try /gauge first, then /counter, /summary, /histogram if no data returns. Important: When PromQL references METRIC_count (the _count component of a histogram), the Cloud Monitoring equivalent is the parent histogram type (/histogram), NOT a separate _count metric. The request count is in the distributionValue.count field of the distribution result. If none return data, list metric descriptors with filter=metric.type = starts_with("prometheus.googleapis.com/METRIC_PREFIX") to find the actual suffix.

Label discovery from PromQL: Cloud Monitoring labels don't always match what you'd guess. Read the PromQL query from policies.json to find the actual label names used. For example, http_server_requests_seconds_count{container="hcs-gb"} tells you the Cloud Monitoring filter needs metric.labels.container = "hcs-gb" — not metric.labels.application or metric.labels.service. Always check the PromQL before constructing filters. PromQL labels on the metric selector map to metric.labels.X in Cloud Monitoring.

Step 2 — Extract threshold from PromQL query text.

When threshold_value is null (PromQL conditions), extract from the query string. PromQL thresholds appear as a comparison operator at the end or after a closing parenthesis: ... > 1000, ... < 0.01, ... == 0. Match with: [><=!]+\s*([\d.]+)\s*$ on the query text.

Mandatory Query Parameters by Metric Class:

Every metric validation query MUST specify ALL THREE aggregation parameters. Omitting crossSeriesReducer when multiple series exist produces sparse, unrepresentative samples.

When in doubt, use REDUCE_MAX for gauges and REDUCE_SUM for counters. Never omit the reducer for metrics that may have multiple series (per-pod, per-container, per-node).

Step 3 — Query the metric.

Use MCP list_time_series (Tier 1) or REST API via temp-file pattern (Tier 2). Select aligner and reducer from the Mandatory Query Parameters table above based on the metric class.

Tier 2 query pattern (uses session temp file per Behavioral Constraint 2):

WORK_DIR="${WORK_DIR:-$(mktemp -d /tmp/ah-XXXXXX)}"
FILTER_FILE=$(mktemp "$WORK_DIR/ts-filter-XXXXXX.txt")
cat > "$FILTER_FILE" << 'FILTER'
metric.type = "prometheus.googleapis.com/jvm_threads_live_threads/gauge"
AND metric.labels.container = "recommendation-service"
AND resource.labels.cluster = "example-cluster-prod"
FILTER

TOKEN=$(gcloud auth print-access-token)
FILTER_ENC=$(python3 -c "import urllib.parse, sys; print(urllib.parse.quote(open(sys.argv[1]).read().strip().replace('\n', ' ')))" "$FILTER_FILE")

# Note: REDUCE_MAX for gauges; use REDUCE_SUM for counters per the table above
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://monitoring.googleapis.com/v3/projects/$PROJECT/timeSeries?filter=$FILTER_ENC&interval.startTime=${START_TIME}&interval.endTime=${END_TIME}&aggregation.alignmentPeriod=3600s&aggregation.perSeriesAligner=ALIGN_MAX&aggregation.crossSeriesReducer=REDUCE_MAX&pageSize=100000" \
  -o "$WORK_DIR/ts-result.json" ; rm -f "$FILTER_FILE"

Write results to $WORK_DIR, not to conversation context. Parse with python3 one-liner to extract p50/p95.

Result Quality Gate — MANDATORY before using any metric result:

After every query, validate the result before computing percentiles:

1. Point count check: For a 14-day window with 1h alignment, expect 336 points per reduced series. If fewer than 50 points are returned, flag the result as insufficient sample and note the actual count. Do NOT use sparse results to derive threshold recommendations — they produce unreliable percentiles. For distribution metrics with daily alignment (14 expected points), use a minimum threshold of 7 points instead of 50.
2. Series count check: After REDUCE_MAX/REDUCE_SUM, expect exactly 1 series. If >1 series is returned, the reducer was not applied — re-query with the correct reducer.
3. Pagination check: If the response contains nextPageToken, the result is truncated. Re-query with higher pageSize or paginate. Do NOT compute percentiles from truncated data.
4. Evidence block logging: Every metric validation result MUST log in the Evidence Ledger: metric type, filter, aligner, reducer, alignment period, series count, point count, and numeric results. This makes cross-report comparisons auditable.

If a query fails the quality gate, mark the evidence basis as heuristic (insufficient sample: N points) instead of measured. Do not promote to Do Now based on insufficient samples.

Step 4 — Compare against threshold.

• If p95 < threshold: the alert rarely fires legitimately. Recommend lowering threshold to p95 + 20% headroom. State: "p95={X}, threshold={Y}, headroom={Z}%".
• If p50 > 0.8 × threshold: baseline crowding. State: "p50={X} is {Y}% of threshold={Z} — baseline crowding".
• Record the query used and numeric result in the cluster's Evidence basis field. Mark as measured.

Step 5 — Handle failures.

If the metric type cannot be mapped, the query returns no data, the suffix is wrong, or the result fails the quality gate: note the reason and keep heuristic basis. A no-data result is itself a finding — the metric may be misconfigured or the resource labels may not match. An insufficient-sample result (< 50 points) means the query parameters need adjustment (missing reducer, low pageSize, or metric sparsity) — do not treat it as measured evidence.

Scope: Query all clusters, not just heuristic-basis. Structural items also benefit from measurement (e.g., confirming queue baselines are zero proves the > 0 threshold fires on transients). If API errors or rate limits occur, prioritize by raw incident count.

Multi-query patterns — these are NOT limitations, they are standard approaches. Use them:

Tier 2 two-query ratio pattern (error rate example):

WORK_DIR="${WORK_DIR:-$(mktemp -d /tmp/ah-XXXXXX)}"

# Query 1: error count (5xx)
FILTER_ERR=$(mktemp "$WORK_DIR/ts-err-XXXXXX.txt")
cat > "$FILTER_ERR" << 'FILTER'
metric.type = "prometheus.googleapis.com/http_server_requests_seconds_count/summary"
AND metric.labels.status = starts_with("5")
AND metric.labels.application = "SERVICE_NAME"
FILTER
# ... curl with crossSeriesReducer=REDUCE_SUM&pageSize=100000 to $WORK_DIR/ts-err-result.json

# Query 2: total count
FILTER_TOTAL=$(mktemp "$WORK_DIR/ts-total-XXXXXX.txt")
cat > "$FILTER_TOTAL" << 'FILTER'
metric.type = "prometheus.googleapis.com/http_server_requests_seconds_count/summary"
AND metric.labels.application = "SERVICE_NAME"
FILTER
# ... curl with crossSeriesReducer=REDUCE_SUM&pageSize=100000 to $WORK_DIR/ts-total-result.json

# Compute ratio
python3 -c "
import json
err = sum(float(pt['value']['doubleValue']) for s in json.load(open('$WORK_DIR/ts-err-result.json')).get('timeSeries',[]) for pt in s['points'] if 'doubleValue' in pt.get('value',{}))
total = sum(float(pt['value']['doubleValue']) for s in json.load(open('$WORK_DIR/ts-total-result.json')).get('timeSeries',[]) for pt in s['points'] if 'doubleValue' in pt.get('value',{}))
print(f'error_rate={err/total*100:.2f}% ({err:.0f}/{total:.0f})') if total > 0 else print('no data')
"
rm -f "$FILTER_ERR" "$FILTER_TOTAL"

If a multi-query approach is available but not executed (e.g., auth expired), state the approach and mark as "not attempted — auth expired" rather than claiming an API limitation.

Stage 4: Coverage Gap Check

Read metric_types_in_inventory from the compute-clusters output. Compare against the Coverage Gap Checklist below. For each gap:

• If the metric type exists in the inventory: check scope — is it applied to all relevant clusters/services, or only a subset? If partial, report "extend scope" with the specific uncovered projects.
• If the metric type does not exist: report "add new".
• Cross-reference each gap with existing noisy clusters that the gap metric would detect upstream (e.g., probe failure coverage → would provide early signal for pod restart clusters).

Routing Validation

Check for routing and ownership gaps using policy-level and cluster-level data from $WORK_DIR.

Zero-channel policies (policy-level): Scan policies.json for entries where notificationChannels is empty AND enabled is true. These fire but notify nobody — invisible failures.

• Policies with raw_incidents > 10 (cross-reference against clusters.json): promote to Investigate with action: "Silent alert — fires but has no notification channels. Add a notification channel or disable."
• Policies with low/zero incidents: surface in Systemic Issues > Dead/Orphaned Config — "Zero-channel policy: {displayName} — enabled but no notification path."

Unlabeled high-noise policies (policy-level): The existing unlabeled_ranking table stays in Systemic Issues. Additionally, for top entries with raw_incidents > 10:

• Promote to Investigate with ownership language: "No squad/team/owner label — ownership is implicit and unauditable. {N} incidents in {days}d have no traceable owner."
• Do not reference org-specific routing fallback channels. The skill is generic.
• Leave suggested owner as "⚠ assign" — the skill does not infer squad ownership from service names.

Label inconsistency promotion (cluster-level): For clusters where label_inconsistency is true AND raw_incidents > 5:

• Promote to Investigate as a cluster-level finding: "Label mismatch — staging/test label on a production resource. {N} incidents may be misrouted or ignored."
• Do NOT merge into the policy-level unlabeled table — entity types stay separate.

SLO Coverage Cross-Reference

Only run this section when $WORK_DIR/slo-source-status.json has "status": "ok". Read $WORK_DIR/slo-services.json (normalized service names) and cross-reference against cluster data.

Service grouping: Group clusters by service_key from clusters.json. Skip clusters where service_key is null. Only consider clusters with signal_family in (error_rate, latency, availability) — exclude other.

SLO migration candidates: For each service_key with total_raw_incidents > 20 across user-facing clusters AND NOT in slo-services.json:

• Surface in Coverage Gaps table: "SLO review candidate — {service_key} has {N} noisy user-facing threshold alerts ({families}), no SLO definition."

SLO review candidates (redundancy check): For each service_key that IS in slo-services.json AND has noisy user-facing threshold alerts:

• Surface as Needs Decision item: "Service {service_key} has an SLO definition and also {N} noisy user-facing threshold alerts; review for redundancy or intentional overlap."
• Do not claim same-signal overlap — slo-services.json carries service names only, not signal coverage metadata.

Matching rules: Both service_key and SLO service names are normalized (lowercase, env suffixes stripped, separators unified). Only exact-match after normalization. Unmatched service_keys are excluded, not fuzzy-guessed.

Stage 5: Produce Report

Write the final report as markdown using the Report Skeleton template below. Group findings by action class (Do Now / Investigate / Needs Decision), not by confidence band. Apply the Do Now Gate to determine which items qualify for the Do Now section. Items that fail the gate drop to Investigate regardless of confidence level.

IaC Location Resolution via Module Discovery

During Do Now gate evaluation, resolve IaC Location for each finding using the iac-modules.json mapping built in Stage 1.

Resolution rules (in priority order):

1. Confirmed: The policy's display_name matches a display_name extracted from a TF file in iac-modules.json (accounting for ${var.XXX} interpolation). The file path is known and verified from the repo tree. Use direct link.
2. Likely: The policy's metric type, PromQL fragment, or resource type matches a known TF module name (e.g., policy using artemis_consumer_count → module message_broker_alerts), but display_name didn't match exactly (e.g., due to complex interpolation). Include the candidate file path.
3. Search Required: No match in iac-modules.json, but {iac_repos} were provided. Include a link to the repo tree (https://github.com/{org}/{repo}/tree/main/tf) for manual search, plus the policy display_name as a search hint.
4. Unknown: No {iac_repos} provided or IaC discovery failed entirely.

Link construction:

• Confirmed/Likely: Direct link to the file: https://github.com/{org}/{repo}/blob/main/{file_path}
• Search Required: Link to the TF directory: https://github.com/{org}/{repo}/tree/main/tf
• If an invocation_file exists (module is instantiated from a separate file with threshold/label variables), include it as a second link: the module definition for the PromQL template, and the invocation file for the configurable values (thresholds, labels, duration).

Do not use gh search code for IaC resolution. Policy IDs are GCP-generated at terraform apply time and never appear in .tf source. GitHub code search for policy IDs always returns zero results.

Prescriptive Reasoning by Pattern

Flapping (raw/episode > 3, raw > 10)

The alert fires and resolves repeatedly on the same resource. Root cause: threshold too close to baseline, or evaluation window too short.

Compare the configured threshold to the median retrigger interval and median duration:

• If median_duration < duration setting: raise duration to 2x median_duration
• If threshold is below observed baseline (from metric validation): raise threshold above p50 of the observed metric. State current vs recommended value.
• If auto_close is NOT SET and median_duration > 1h: set auto_close to 2x median_duration (capped at 86400s / 24h)
• If notification_channels > 0 and noise_score >= 5: recommend demoting to dashboard-only
• If raw/episode ratio > 10: the alert has a structural flap design — recommend auto_close or redesign

Chronic (episodes >= 5, median_duration > 1h)

The alert fires and stays open for extended periods. The underlying issue is real and persistent.

This is "Fix the underlying issue". The alert is working correctly — the service has a problem:

• State specific investigation pointers based on metric_type and resource_type
• If auto_close is NOT SET: recommend setting it to match the evaluation window to prevent alert accumulation
• Do not recommend raising the threshold — the alert is detecting a real condition
• If distinct_resources is high (> 20): flag as systemic/cluster-wide, not single-service

Recurring (episodes >= 5, median_duration < 1h)

The alert fires regularly but resolves within an hour. Could be deploy-time transients, batch jobs, or genuine intermittent issues.

• If tod_pattern shows > 50% concentration at a specific hour: infer scheduled job or deployment window. Recommend time-of-day mute for that window or extend duration to ride through the transient. Flag as low confidence if < 80% concentration.
• If

…(truncated)