# Incident Analysis

> Use when investigating production symptoms — connection failures, pod crashes/restarts, SIGTERM/OOM errors, latency spikes, Cloud SQL/proxy issues, deployment-correlated errors, ImagePullBackOff, CreateContainerConfigError, or node NotReady events

- Skill: `dkpapadopoulos/incident-analysis` (Agent Skill, multi-file: 26 files)
- Install (CLI): `npx skillmds@latest add dkpapadopoulos/incident-analysis`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dkpapadopoulos/incident-analysis/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: dkpapadopoulos (https://skillmd.com/u/dkpapadopoulos)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/dkpapadopoulos/incident-analysis

---


# Incident Analysis

Tiered GCP log investigation with playbook-driven mitigation and structured validation. Stages: (opt-in) INTAKE → MITIGATE → CLASSIFY → INVESTIGATE → EXECUTE → VALIDATE → POSTMORTEM → (opt-in) REPORT-BACK. Detects available tools at runtime and uses the best tier. Playbook YAML files define mitigation commands, safety invariants, and validation criteria.

## Stage Flow

```dot
digraph stages {
    rankdir=LR;
    node [shape=box];
    INTAKE -> MITIGATE [label="opt-in\nJira ticket\n(or skip)"];
    MITIGATE -> CLASSIFY [label="mitigation\nneeded\n(Step 5)"];
    MITIGATE -> INVESTIGATE [label="no mitigation\n(Step 6)"];
    CLASSIFY -> EXECUTE [label="high\nconfidence"];
    CLASSIFY -> INVESTIGATE [label="low\nconfidence\n(Steps 1-5 only)"];
    INVESTIGATE -> CLASSIFY [label="re-classify\nafter probes"];
    INVESTIGATE -> POSTMORTEM [label="completeness\ngate passed"];
    EXECUTE -> VALIDATE;
    VALIDATE -> POSTMORTEM [label="success"];
    VALIDATE -> INVESTIGATE [label="failed"];
    POSTMORTEM -> REPORT_BACK [label="opt-in\nJira comment"];
}
```

Key re-entry paths:
- **CLASSIFY < 60 → INVESTIGATE:** Only Steps 1–5 run. Steps 6–9 skipped. Findings feed back to CLASSIFY.
- **VALIDATE failed → INVESTIGATE:** Full Stage 2. The mitigation didn't work.

## Quick Reference — Symptom to Playbook

| Symptom | Likely Playbook | Category |
|---------|----------------|----------|
| Error spike after deploy | bad-release-rollback | bad-release |
| CrashLoopBackOff / pod restarts | workload-restart | workload-failure |
| Multi-pod probe timeout on one node | node-resource-exhaustion | infra-failure |
| Node NotReady / kubelet down | infra-failure | infra-failure |
| CPU/memory saturation + traffic spike | traffic-scale-out | resource-overload |
| Config change correlated with errors | config-regression | bad-config |
| Upstream dependency errors | dependency-failure | dependency-failure |

**When NOT to use this skill:**
- Capacity planning or resource right-sizing without an active incident — use proactive monitoring tools instead
- Alert tuning or alert hygiene — use the `alert-hygiene` skill
- Non-production debugging (local dev, CI failures) — use `systematic-debugging`
- Performance optimization without user-facing symptoms — not an incident

## Behavioral Constraints (Always Active)

### 1. HITL Gate — No Autonomous Mutations

If a mutating action is identified (restart service, rollback deployment, scale pods, modify config), you MUST present the exact command you intend to run and HALT completely. Wait for explicit user confirmation before executing. Prefix any such command with a `RISK:` label (`references/command-risk.md`); read-only queries are never labeled.

### 2. Scope Restriction — No Global Searches During Incidents

During active investigation, all application-level file reads, log queries, and code searches MUST be constrained to the specific service or trace ID identified in Stage 1 (MITIGATE). Global codebase searches (unbounded grep, recursive find) are forbidden. This prevents context window exhaustion and irrelevant noise during time-sensitive debugging.

**Bounded exceptions:** (a) Infrastructure escalation — when Step 3 identifies multi-pod failures indicating a node-level root cause, scope expands to the affected node(s) and their infrastructure signals. The completeness gate (Step 8, Q6) may require checking peer nodes. (b) Shared resource — when Step 2 identifies Tier 1 errors in adjacent services, or Step 3 identifies a shared resource under pressure, scope expands to the shared resource's known consumer set. Both escalations are bounded to specific implicated targets, not the entire cluster or organization.

### 3. Temp-File Execution Pattern (Tier 2 Only)

For any LQL query longer than 5 words or containing quotes/regex, write the query to a session-scoped temp file via `mktemp` and execute via file read. This avoids escaping failures and concurrent-session race conditions:

```bash
LQL_FILE=$(mktemp /tmp/agent-lql-XXXXXX.txt)
cat > "$LQL_FILE" << 'QUERY'
resource.type="cloud_run_revision"
AND resource.labels.service_name="checkout-service"
AND severity>=ERROR
AND timestamp>="2026-03-19T10:00:00Z"
QUERY

gcloud logging read "$(cat "$LQL_FILE")" \
  --project=my-project --format=json --limit=50 ; rm -f "$LQL_FILE"
```

The `;` operator ensures cleanup runs regardless of whether `gcloud` succeeds or fails. `mktemp` with a random suffix prevents concurrent sessions from overwriting each other's queries.

### 4. Context Discipline on Stage Transitions

Claude cannot literally clear its context window mid-session. This constraint is enforced **behaviorally** through prompt instructions:

When transitioning from INVESTIGATE to POSTMORTEM:
1. Write a synthesized summary of the timeline and root cause as an explicit output block
2. From that point forward, you are **strictly forbidden from referencing the raw JSON log outputs** from earlier in the conversation
3. Draft the postmortem **ONLY from the synthesized summary**
4. No further log queries or source code reads are permitted during POSTMORTEM

### 5. Evidence Freshness Gate

If the user has not approved a mitigation proposal within the playbook's `freshness_window_seconds` of evidence collection, the proposal is retracted. Return to CLASSIFY with fresh queries. Stale evidence cannot be acted upon.

### 6. Evidence Ledger — Reuse Within Freshness Window

Maintain a mental ledger of evidence collected during the investigation, keyed by query fingerprint. The key must include enough dimensions to prevent collapsing different namespaces, trace-scoped vs service-scoped reads, or caller-vs-affected-service queries. The fingerprint varies by query type to prevent false collisions:

| Query type | Fingerprint key |
|-----------|----------------|
| Log queries (LQL) | `(service, environment, severity, LQL_filter_hash, time_window)` |
| Metric queries | `(service, environment, metric_type, aggregation, time_window)` |
| kubectl queries | `(resource_kind, name, namespace, context, output_format)` — e.g. `get deployment/X -o json` vs `rollout history` vs `jsonpath={.status.readyReplicas}` are distinct entries |
| Trace queries | `(trace_id, project_id)` |
| Source analysis | `(repo, commit_ref, file_path)` |

Before issuing a query matching a prior entry: reuse if within `freshness_window_seconds` (default 300s), labeling output as `reused (collected at <UTC>)`. If stale, re-query and update.

**Mandatory re-query exceptions:** EXECUTE fingerprint recheck, VALIDATE sampling, and user-requested fresh data — always re-query live state.

### 7. Evidence-Only Attribution — No Speculative Causal Claims

Every causal claim in synthesis, YAML, and postmortem must reference a specific query result. Words like "likely", "probably", "possibly" are prohibited in final attribution — replace with evidence-backed language (`"caused by X (evidence: [result])"`) or move to `open_questions`. Speculative language IS permitted in intermediate notes where it drives the next query.

**Self-check:** Before emitting the Step 7 synthesis, scan for "likely", "probably", "possibly", "presumably", "may have", "might be" in causal sentences. Replace each with evidence-backed language or move to `open_questions`.

### 8. MCP Result Processing — Never Re-Parse Cached Files

MCP tool results (especially `list_log_entries` and `list_time_series`) can exceed Claude Code's result caching limit (~100K characters). The `tool-results/` file on disk will contain truncated JSON — reading it via `cat ... | jq`, `python3 json.loads(open(...))`, or similar will fail with a parse error.

Note: This constraint applies to on-disk `tool-results/` files written by the Claude Code harness, not to the Evidence Ledger (Constraint 6). Reusing results held in-context is still governed by Constraint 6's freshness rules.

**Rules:**

1. **Extract needed fields in the same turn the MCP tool returns.** Summarize timestamps, severity, error messages, trace IDs, and resource labels directly from the tool response. Do not defer processing to a later Bash command that reads the cached file.

2. **Never read `tool-results/` files.** If you find yourself writing `cat tool-results/...`, `json.load(open('tool-results/...'))`, or piping a cached MCP result through jq — STOP. Re-invoke the MCP tool with a smaller `page_size` instead.

3. **If a single MCP response is too large to process inline**, re-query with `page_size` halved (50 → 25 → 10) requesting only needed fields. At `page_size=10`, fall back to Tier 2 using Constraint 3's temp-file pattern.

4. **For multi-step processing of the same result set** (e.g., fingerprint then exemplar extraction), summarize the result into a compact intermediate form (list of `{timestamp, severity, message_prefix, trace_id}` objects) in the same turn the MCP tool returns. Reference the summary in subsequent steps — not the raw result and not a cached file.

**Self-check:** Before issuing any Bash command, check whether the command path contains `tool-results/`. If it does, stop and re-invoke the MCP tool with a smaller `page_size` instead.

### 9. Intermediary-Layer Investigation Discipline

When the symptomatic entry point is an intermediary — reverse proxy, API gateway, service mesh sidecar, message broker, or load balancer — errors observed at that layer describe **where** the system broke but not **why**. Before forming a root cause hypothesis (Step 5), identify and query every distinct downstream service that appears in the intermediary's error logs.

**Scope boundary:** This sweep is bounded to services explicitly named in the intermediary's error output. It does not authorize cluster-wide searches.

Step 3c codifies this sweep as a structured procedure with inventory output.

### 10. Dual-Layer Investigation

For every service in the error chain, the investigation must assess both the infrastructure layer (deployment history, pod state, resource pressure) and the application layer (exception class, error mechanism). Neither layer alone is sufficient to close an investigation.

**Minimum per-service evidence (enforced via Step 3c):**
- **Infrastructure:** 72-hour deployment history + at least one runtime signal (pod state/events, resource metrics, or error rate trend)
- **Application:** the service's own ERROR logs queried, dominant exception/error class identified, mechanism status recorded as `known` (traced to code path, cache state, or consumer behavior) or `not_yet_traced` (error class known, mechanism not investigated)

**Full mechanism-level depth** is mandatory for:
- The chosen root-cause service (must trace to specific code path, cache/config state, retry/amplification behavior, or consumer mechanism)
- Any service that triggers Step 3c → Step 3 re-entry (see Step 3c escalation rule)

For all other services, mechanism status `not_yet_traced` is acceptable — it records that the error class is known but the application-layer mechanism was not deeply investigated. This prevents over-investigation of obvious victims while ensuring the root-cause service is traced to mechanism.

**Anti-pattern this prevents:** Building a complete, internally-consistent infrastructure narrative (timeouts, resource pressure, GC pauses) while the actual root cause is an application-layer bug (stale cache, template error, retry storm) in a service whose ERROR logs were never queried.

### 11. Intermediate Conclusion Verification

Any intermediate conclusion that will be used in the causal narrative must be explicitly stated and tested with at least one disconfirming query before building on it. This applies to conclusions formed during any investigation step, not just the final hypothesis.

**Common intermediate conclusions that require verification:**
- "This error is baseline noise" → query the baseline rate and compare numerically (see Tier 3 verification rule)
- "This service is healthy / not involved" → query its own ERROR logs in the incident window
- "This failure is dependent on the primary root cause" → verify the service's error class matches the hypothesized mechanism
- "This workload is the trigger" → check whether it ran without incident on the previous cycle (see recurring-workload trap)
- "This service's 403/500 responses are expected" → verify the response rate against a non-incident baseline

**Self-check:** Do not build the next investigation step on a conclusion that was inferred but not queried. If you catch yourself thinking "this is probably X" without having queried for confirmation, stop and query.

### 12. Evidence Links

For each claim surface in the Step 7 synthesis — chosen root-cause statement, each ruled-out hypothesis, and each `service_error_inventory` entry — include clickable verification links when URL parameters were captured at query time. This constraint is active across Steps 2-7.

**Caps:** max 3 links per root cause, max 2 per ruled-out, max 3 per inventory entry. Omit when URL parameters are unrecoverable. Never emit placeholder, reconstructed, or guessed URLs.

**Capture rule:** Record link inputs (project_id, LQL filter, time window, trace_id, commit SHA, metric_type) at query time. Retroactive construction is permitted only when exact original query parameters are visible verbatim in the conversation — never from prose summaries or inferred values.

Full specification — link types, YAML shape, priority rule, omission rules, label normalization, URL templates: `references/evidence-links.md`.

### 13. Parallel Execution Strategy — Batch Independent Queries

Full detail: see `references/parallel-execution.md`.

## Investigation Modes

### Default: Full Investigation

The complete 6-stage pipeline with all steps executed in order: access gate, full inventory, impact quantification, aggregate fingerprint, investigation, classification, and postmortem. This is the default for `/investigate` and all incident-analysis activations.

### Opt-In: Live Triage

An explicit fast path that prioritizes time-to-first-hypothesis for active, ongoing incidents. Activated only when the user explicitly requests it (e.g., "quick triage", "what's happening right now", "live triage"). The skill may suggest this mode when the prompt describes an active incident, but must not silently switch into it.

**Live-triage behavior — what changes:**

| Step | Full Investigation | Live Triage |
|------|-------------------|-------------|
| Step 1b (Access Gate) | Blocking — wait for fix or explicit proceed | Non-blocking — snapshot access state, proceed immediately, note gaps |
| Step 2b (Inventory) | Full — replicas, distribution, resources, probes, scheduling | Light inventory only — replica count and current status (one query). Deep inventory deferred until after first hypothesis or when symptoms indicate node/distribution dependence |
| Step 2c (Impact) | Before first log query | Deferred until after first hypothesis |
| Steps 3-4 | Unchanged | Unchanged |
| Steps 5-9 | Unchanged | Unchanged |
| CLASSIFY/EXECUTE/VALIDATE | Unchanged | Unchanged — fingerprint recheck, completeness gate, and all safety rules apply |
| POSTMORTEM | Unchanged | Unchanged |

**What does NOT change in live-triage:**
- Access state is still recorded for evidence_coverage
- Light inventory still captures replica count (prevents gross mis-scoping)
- The completeness gate still runs — deferred steps are flagged as gaps if never backfilled
- EXECUTE fingerprint recheck is never skipped
- HITL gate for mutations is never skipped

**Mode recorded in synthesis:** The `investigation_summary.scope.mode` field captures which mode was used, so the postmortem and completeness gate know whether deferred steps were intentional.

## INTAKE (opt-in Jira ticket)

Create or adopt a Jira ticket before investigation begins. This stage is opt-in: it activates only when the user explicitly requests a ticket (e.g. "file a Jira ticket") or supplies a ticket key (e.g. "investigate NC-1234"). When neither signal is present, skip INTAKE and begin at Stage 1. On creation, present the exact ticket payload and HALT for approval before calling `createJiraIssue`. Full procedure: `references/jira-intake.md`.

## Stage 1 — MITIGATE

### Step 1: Detect Available Tools

Run the shared observability preflight to check environment readiness:

```bash
bash "${CLAUDE_PLUGIN_ROOT}/scripts/obs-preflight.sh"
```

Parse the JSON output to select the execution tier:

**Tier 1 — MCP (`@google-cloud/observability-mcp`):**
If you have access to `list_log_entries`, `search_traces`, `get_trace`, `list_time_series`, or `list_alert_policies` as MCP tools in this session, **verify auth before classifying as Tier 1**. Make a lightweight probe call (e.g., `list_log_entries` with `pageSize=1` and a narrow 1-minute window) to confirm the tools return data rather than auth errors.

**If the probe fails with an auth error** (`invalid_grant`, `invalid_rapt`, `UNAUTHENTICATED`, `token expired`):
1. Report the specific error to the user
2. **Immediately offer the fix** — do NOT silently fall back to Tier 2:

   > "MCP Observability auth expired (`<error>`). Tier 1 provides metrics, traces, and error reporting that gcloud CLI cannot. Fix now with:
   > ```
   > ! gcloud auth application-default login
   > ```
   > This opens a browser for re-authentication. Proceed?"

3. If the user re-authenticates, re-probe to confirm, then classify as Tier 1
4. **Only fall back to Tier 2 if the user explicitly declines** (e.g., "skip it", "proceed without", "no time"). Record the declined fix in the access gate for `evidence_coverage`.

**Why this matters:** Tier 2 (gcloud CLI) cannot do `list_time_series` (metrics), `get_trace` (trace correlation), or `list_group_stats` (error reporting). Silently falling back to Tier 2 creates investigation gaps that are expensive to discover later — especially for database metrics, which are often the missing piece in shared-resource incidents. The few seconds to re-authenticate are far cheaper than the gaps.

**Tier 2 — gcloud CLI via Bash:**
```bash
command -v gcloud && gcloud logging read --help >/dev/null 2>&1 && echo "gcloud: available" || echo "gcloud: not available"
```
If gcloud is available but not authenticated, guide through `gcloud auth login` and `gcloud auth application-default login`.

**Tier 3 — Guidance-only:**
If neither MCP tools nor gcloud are available, provide manual Cloud Console instructions (Logs Explorer URL patterns, filter syntax).

**Tier upgrade nudge:** If using Tier 2 (gcloud CLI) and Tier 1 MCP tools are not available, include a one-line note after reporting the tier:

> "Using gcloud CLI (Tier 2). For faster queries with autonomous trace correlation, run `/setup` to configure GCP Observability MCP (Tier 1)."

Do not repeat this nudge after the first mention.

### Step 1b: Access Gate — Fix Before Proceeding

After detecting the execution tier, present a tool access summary and prompt the user to fix **fixable** gaps (expired auth, wrong context) before continuing. Do not block on unfixable gaps (tool not installed) during an active incident.

```
Tool access:
  MCP observability: available | auth expired (fixable) | unavailable
  gcloud auth:       active (project X) | expired | not configured
  kubectl context:   available (context Y) | unavailable
  GitHub CLI (gh):   authenticated | not authenticated | not installed

Investigation domains:
  Logs:              ✓ complete (Tier N) | ⚠ partial | ✗ unavailable
  Metrics:           ✓ complete (Tier N) | ⚠ partial (Tier 2 — no list_time_series) | ✗ unavailable
  K8s state:         ✓ complete | ✗ unavailable
  Source analysis:    ✓ complete | skipped | ✗ unavailable
  Trace correlation: ✓ complete (Tier 1 only) | skipped | ✗ unavailable
```

**MCP auth expired is always a fixable gap.** When the access gate shows `auth expired (fixable)`, the Metrics and Trace correlation domains will show degraded or unavailable. Present this prominently:

> "⚠ MCP auth expired — Metrics (`list_time_series`), Traces (`get_trace`), and Error Reporting (`list_group_stats`) will be unavailable. Fix with `! gcloud auth application-default login`?"

**If any other fixable gap is detected**, present the fix command and ask:

> "⚠ [Domain] will be unavailable without [tool/auth]. Fix now, or proceed with degraded access?"

Wait for the user to fix or explicitly proceed. Record the access state — including whether a fix was offered and declined — for the `evidence_coverage` block in Step 7.

### Step 2: Establish Scope

Identify:
- Which service?
- Which environment (production, staging)?
- What time window? (default: last 30-60 minutes). If the user provides a local time (e.g., "it broke at 2pm"), convert to UTC using the session's timezone (`date +%z`) before querying. If the session timezone cannot be determined, ask the user. All subsequent timestamps in the investigation and postmortem MUST be in UTC.

### Step 2b: Establish Inventory

Before querying logs, determine what you are investigating:
- How many replicas/instances exist? (query metrics or deployment spec — do not infer from logs)
- Where are they distributed? (nodes, zones, regions)
- What are the resource requests, limits, and probe configurations?
- For k8s workloads: scheduling constraints — pod affinity/anti-affinity rules, topologySpreadConstraints, node affinity, taints and tolerations. Query from the same deployment spec used for resource requests. If kubectl is unavailable, check GitOps manifests via `gh api` or `git show` as fallback.

This prevents scoping errors (investigating 4 pods when 7 exist) and reveals distribution risks (3 of 7 pods on one node) before they become surprises in the postmortem. For k8s, use `container/memory/request_bytes` grouped by pod name. For other platforms, use the equivalent inventory query.

Note: topologySpreadConstraints and podAntiAffinity both control pod distribution — if either is present, the workload has scheduling constraints. Distinguish enforcement level: soft (ScheduleAnyway, preferredDuringScheduling) vs hard (DoNotSchedule, requiredDuringScheduling).

When live cluster access is unavailable for inventory or action item verification, fall back to GitOps manifests (`gh api repos/ORG/REPO/contents/PATH` or `git show`) to check deployment configuration. If neither is available, flag affected inventory fields and action items as unverified.

### Step 2c: Quantify User-Facing Impact

Before diving into root cause, establish the impact magnitude from available sources:
- **From metrics (query):** HTTP 5xx **and 4xx** error count/rate at the load balancer or ingress (a persistent per-user failure is often a **404 for a not-yet-provisioned resource**, not a 5xx), SLI degradation (latency, availability), affected endpoint paths.
- **From alerts (check):** If an SLO burn rate alert fired for this service in the time
  window, note the alert name, burn rate value, and error budget remaining. This provides
  severity context before the deep dive. Check via `list_alert_policies` (Tier 1) or
  `gcloud alpha monitoring policies list` (Tier 2).
- **From user-provided context (do not query):** support tickets, user reports, business impact descriptions. Incorporate if provided but do not attempt to query external support/ticket systems.
- **If neither is available:** state "user-facing impact not quantified" and proceed. Do not estimate.

This frames severity before the deep dive — a 1,100-error incident gets different treatment than a 5-error incident.

**Clean 5xx/ERROR sweep ≠ backend healthy.** `severity>=ERROR`/`status>=500` both exclude **4xx**; a persistent per-user failure with a clean 5xx sweep is often a **404 for a not-yet-provisioned resource**, visible only at the gateway/access-log layer or an app-level status field. See `references/4xx-sweep-blind-spot.md`.

### Step 2d: Baseline-First Gate — Skip Baseline Signals Early

Before deep-diving into any error signal, compare its count against a baseline from a non-incident period (same service, same error class, same time-of-day window on a prior day — preferably the same weekday).

**Decision rule:**
- **count_incident < 1.5 × count_baseline** → **baseline** — skip. Do NOT deep-dive. Record in the synthesis as `"baseline — skipped (N incident vs M baseline)"` and move on.
- **1.5× ≤ count_incident < 10× count_baseline** → **elevated** — proceed with investigation but note the baseline for context.
- **count_incident ≥ 10× count_baseline** → **anomalous** — prioritize for immediate deep-dive.
- **count_baseline = 0 and count_incident > 0** → **new** — always investigate.

These are rate-based classifications independent of the Step 2 error taxonomy. A Tier 2 infrastructure error at baseline rate is still classified as **baseline** and skipped.

**When to apply:** This gate applies at two points:
1. **Step 3 (initial error query)** — before selecting which error signals to pursue in Steps 3b/4
2. **Step 3c (multi-service sweep)** — before deep-diving into any service's errors (item 2 in the procedure)

**Why this matters:** Without this gate, the investigation will deep-dive into every error signal regardless of whether it's normal. In one investigation, ~4 query round-trips were spent investigating 403 errors that turned out to be at baseline rate (4,948 vs 5,000) — time wasted that could have been spent on the actual anomaly (JDBC errors: 500 vs 0 baseline).

**Implementation:** Query the incident count and baseline count **in parallel** (two queries in the same batch). Compute the ratio before proceeding. This adds one query round-trip but saves many by eliminating baseline signals early.

### Step 3: Query Error Rate / Recent Errors

Scoped to the identified service + narrow time window.

**Tier 1:** Use `list_log_entries` with LQL filter scoped to service + severity + time window, `page_size` <= 50.

**Tier 2:** Use the temp-file execution pattern (see Constraint 3) with `gcloud logging read` and `--limit=50`.

### Step 3b: Aggregate Error Fingerprint

Before reading raw log entries for pattern extraction, query the error distribution to identify the dominant error class. This prevents sample bias — 50 recent entries can overrepresent the latest error class instead of the most frequent one.

**Preferred path — Error Reporting API (identifies error signatures):**

**Tier 1:** If `list_group_stats` is available as an MCP tool, use it with the service's project_id and the investigation time range. This provides server-side grouping by recurring error signature with counts.

**Tier 2:** If gcloud is available, try `gcloud beta error-reporting events list --service=<service> --format=json --limit=20`. Same backend, same output.

**If Error Reporting is unavailable** (API not enabled, tool not present, or service doesn't emit structured errors):

**Tier 1/2 fallback — severity counts:** Query `list_time_series` with `logging.googleapis.com/log_entry_count` to get error volume by severity and container. This answers "how many errors?" but not "which error classes?" — use it for magnitude only.

**Tier 2 fallback — client-side bucketing:** Fetch up to 100 log entries (2× the normal Step 3 sample) and group by error message prefix (first 80 chars of `jsonPayload.message` or `textPayload`). **⚠ This is sample-biased** — label results as `aggregation_source: sample` in the synthesis.

**If no aggregate source is available at all:** Proceed with Step 3's existing 50-entry sample. Note `aggregation_source: unavailable` in the synthesis and record as a gap.

**Output:** Identify the top 3-5 error buckets by frequency. Record:
- `aggregation_source`: `error_reporting` (signature-grouped), `metric` (severity counts only), `sample` (client-side bucketing), or `unavailable`
- Dominant bucket with count/percentage
- Whether dominance is clear (>50% of errors) or ambiguous (no bucket >30%)

**Step 4 then becomes exemplar-driven:** Fetch 3-5 raw log entries per dominant bucket for detailed analysis (stack traces, request IDs, trace IDs). Raw logs are exemplars for known buckets, not the discovery mechanism.

### Step 4: Identify Failing Request Pattern (Exemplar-Driven)

Using the dominant error buckets from Step 3b, fetch 3-5 raw log entries per top bucket as exemplars. Extract: endpoint, error code, stack traces, request/trace IDs. If Step 3b was skipped (aggregate tools unavailable), fall back to the current behavior: extract patterns from the Step 3 sample, but note `aggregation_source: unavailable` in the synthesis.

### Step 5: Mitigation Routing

If mitigation is needed, transition to CLASSIFY for structured playbook selection. All mutating actions must go through the playbook framework — the agent cannot propose bare commands outside the safety contract. If no playbook matches, transition to INVESTIGATE or provide manual guidance.

### Step 6: Transition

If a code fix is needed (no mitigation required), transition to INVESTIGATE.

## CLASSIFY

Structured playbook selection. Entered from MITIGATE Step 5 when mitigation is needed. Evaluates signals against candidate playbooks, scores them, and routes to the appropriate confidence tier.

### Playbook Discovery

Load candidate playbooks from two sources:

1. **Bundled playbooks:** `skills/incident-analysis/playbooks/*.yaml` (shipped with the plugin)
2. **Repo-local overrides:** `playbooks/incident-analysis/*.yaml` (project-specific)

Resolution is by `id` — a repo-local playbook with the same `id` as a bundled playbook replaces the bundled version entirely. Repo-local playbooks with unique IDs are added to the candidate set.

### Signal Evaluation

For each signal referenced by candidate playbooks, evaluate against current evidence to produce a tri-state result:

| State | Meaning |
|-------|---------|
| `detected` | Signal is present in the collected evidence |
| `not_detected` | Signal was explicitly looked for and is absent |
| `unknown_unavailable` | Cannot evaluate (tool unavailable, data not collected, ambiguous) |

**Compound signal propagation:**
- `any_of`: detected if ANY child is detected; not_detected if ALL children are not_detected; unknown_unavailable otherwise
- `all_of`: detected if ALL children are detected; not_detected if ANY child is not_detected; unknown_unavailable otherwise

Signal definitions are loaded from `skills/incident-analysis/signals.yaml`.

### Scoring, Routing, and Decision Records

Scoring formula, confidence-gated routing (high/medium/low), loop termination, disambiguation anti-looping, and decision record templates are in `references/classify-scoring.md`.

**Quick summary:** Each playbook scores `confidence = clamp(0,100, round(base_score - contradiction_score) / evaluable_weight × 100)`. Veto signals disqualify immediately. Coverage gate requires evaluable_weight/max_possible ≥ 0.70. High confidence (≥85 with ≥15pt margin) → HITL gate. Medium (60-84) → disambiguation probes then re-rank. Low (<60) → INVESTIGATE Steps 1-5 then re-classify.

## Stage 2 — INVESTIGATE

**Re-entry from CLASSIFY (< 60 path):** When entered from the CLASSIFY low-confidence path, only Steps 1-5 run. Steps 6-9 (Flight Plan, Timeline Extraction, context synthesis, completeness gate, POSTMORTEM transition) are SKIPPED. Findings feed back to CLASSIFY for reclassification.

### Step 1: Query Logs with Narrowed Filter

Use LQL scoped to service + severity + time window identified in Stage 1.

### Step 2: Extract Key Signals

Stack traces, error messages, request IDs, trace IDs.

**Error taxonomy — prioritize by diagnostic value:** Classify signals into Tier 1 (anomalous — trigger indicators), Tier 2 (infrastructure — where it's breaking), Tier 3 (expected — at verified baseline rates). Investigate Tier 1 first. Message broker signals are always Tier 1 — trace to consumer's exception before investigating downstream infrastructure symptoms. Tier 3 requires a verified baseline rate comparison; "this looks like it always happens" is not evidence — query the baseline rate. Container exit codes (0, 1, 137/OOMKilled, 139/SIGSEGV, 143/SIGTERM) guide investigation routing. Full taxonomy, exit code guide, and routing rules: `references/error-taxonomy.md`.

### Targeted Disambiguation Probes (Conditional — CLASSIFY Handoff)

When entered from the CLASSIFY low-confidence path with a SHORTLIST handoff artifact,
execute the disambiguation probes listed for each runner-up. Probes are read-only,
aggregate-first queries (max_results <= 10) that target signals specific to the runner-up
playbook. Each probe runs once per classification fingerprint.

**Scope exception:** Disambiguation probes may query declared/known dependencies of the
affected service, within the same time window as the primary investigation
(e.g., upstream APIs, backing datastores) even though those are technically
outside the single-service scope restriction. This is permitted because playbook
`disambiguation_probe` definitions are pre-authored and bounded — they cannot expand into
unbounded global searches.

After all probes complete, feed results back to CLASSIFY for reclassification.

### Step 3: Single-Service Deep Dive

- Error grouping (frequency, first/last occurrence)
- Recent deployment correlation (deploy timestamp vs. error spike?)
- Resource metrics (CPU, memory, latency) if available
- **Database/connection-pool metrics (conditional — when JDBC or connection errors are present):**

  When the service shows `JDBCConnectionException`, `acquisition timeout`, `pool exhausted`, `ConnectionRefused` to a database, or Cloud SQL proxy errors, **query the database's own metrics before attributing the issue to database capacity**. This is mandatory — do not conclude "database under pressure" or "connection starvation" from application-side errors alone.

  **Required queries (incident window + baseline, in parallel per Constraint 13):**

  | Metric | Tier 1 (`list_time_series`) | Tier 2 (REST API via `curl` + `gcloud auth print-access-token`) |
  |--------|----------------------------|----------------------------------------------------------------|
  | Connection count | `cloudsql.googleapis.com/database/network/connections` | Same metric via Monitoring REST API |
  | CPU utilization | `cloudsql.googleapis.com/database/cpu/utilization` | Same metric via Monitoring REST API |
  | Query rate | `cloudsql.googleapis.com/database/mysql/questions` (MySQL) or `cloudsql.googleapis.com/database/postgresql/transaction_count` (PostgreSQL) | Same metric via Monitoring REST API |

  **Decision branch:**

  | DB metrics vs baseline | Diagnosis | Investigation route |
  |------------------------|-----------|-------------------|
  | Connections, CPU, query rate all **normal** (within 1.5x baseline) | **App-side pool exhaustion** — connections held too long, not too many | Investigate application: slow queries holding connections, transaction scope changes, connection leak, pool sizing (max size, timeout, leak detection). Check deployment history for code changes affecting connection lifecycle. |
  | Connections **elevated** (approaching or at `max_connections`) | **Database-level exhaustion** — too many consumers | Investigate database: `max_connections` setting, per-service pool sizing, total connection demand across all consumers. Consider connection isolation (dedicated instances for critical services like auth). |
  | CPU **elevated** (>80%) or query rate **spiked** | **Database under load** — slow queries or query volume | Investigate database: slow query log, query plan regressions, lock contention. Check for new query patterns from recent deployments. |

  Rows are not mutually exclusive. When multiple conditions are present (e.g., connections elevated AND CPU spiked), combine diagnoses: the database is both oversubscribed and overloaded.

  **Anti-pattern this prevents:** Concluding "shared database starvation" from application-side JDBC errors when the database itself is healthy. In one investigation, this led to incorrect action items ("isolate auth service DB") that were revised after database metrics showed normal connection count, CPU, and query rate — the issue was app-side pool exhaustion (connections held too long under normal DB load).

- **Application-logic analysis (for the dominant error path):**
  - **Call pattern detection:** From stack traces in Step 2 exemplars, determine whether the failing code path makes sequential (N+1) calls to the degraded dependency. A loop calling `checkPermission()` per item is N+1; a single `batchCheck()` call is not. If N+1 is detected, note the amplification factor (items per request x latency per call = total request latency).
  - **Retry/amplification analysis:** Check whether the calling code retries failed requests. If a 3-second timeout triggers a retry, each retry adds 3 more seconds of dependency pressure. Look for retry configuration in the stack trace's framework (e.g., Camel redelivery, Spring Retry, gRPC retry policy).
  - **gRPC connection distribution (conditional — when dependency uses gRPC):** If server-side logs include `peer.address`, sample 1 minute of calls and group by caller IP. Compare the distribution against the expected even split (1/N where N = number of client pods). If one caller's share is disproportionately high relative to the expected baseline, flag as potential connection pinning (HTTP/2 over K8s Service ClusterIP load-balances at connection level, not request level). Note: there is no universal threshold — what matters is whether the skew is large enough to explain the observed latency. Report the actual distribution and let the investigator judge.

**CrashLoopBackOff triage (conditional — when `crash_loop_detected` signal is present):** Complete the diagnostic sequence in `workload-restart` playbook's `investigation_steps` before proposing restart: pod describe → events → termination reason and exit code → previous container logs → deployment/probe config → rollout history correlation. Redirect to other playbooks when evidence warrants (OOMKilled → resource, stack trace after deploy → bad-release, ImagePullBackOff/CreateContainerConfigError → pod-start failure). Full CrashLoopBackOff triage, probe/startup-envelope checks, and Pod-start failure branch: `references/deep-dive-branches.md`.

**Infrastructure escalation (conditional):** If Step 3 reveals that multiple pods or services are failing simultaneously — especially with `context deadline exceeded`, widespread probe timeouts, or errors localized to a single node — verify whether the root cause is at the node or infrastructure level by checking:
- Node resource metrics (memory/CPU allocatable utilization)
- kubelet logs (housekeeping delays, probe failures, eviction events)
- GCE serial console (kernel OOM, balloon driver, memory pressure)
- Audit logs (maintenance-controller, drain events)

If a `node-resource-exhaustion` playbook is available, transition to CLASSIFY for structured scoring.

**Shared resource escalation (mandatory when detected):** If the degraded service is used by multiple consumers (authorization service, database, message broker, cache cluster, shared API gateway), follow the caller investigation procedure in `references/caller-investigation.md`. This is mandatory, not optional. The procedure identifies dominant callers, checks their ERROR logs and deployment history, compares distribution to baseline, and checks for amplification loops. Bounded to the shared resource's known consumer set — no unbounded global searches.

### Step 3c: Multi-Service Error Sweep

**Gate:** This step is mandatory when the error chain involves 2 or more services — whether discovered through an intermediary layer (Constraint 9), trace correlation (Step 4), shared-resource escalation (Step 3), or proxy error logs. Skip only for confirmed single-service incidents with no cross-service error signals.

**Parallel sweep pattern (Constraint 13):** When the error chain involves 3+ service

…(truncated)