# GCP Debug

> Diagnose latency, errors, freezes and scaling issues across Cloud Run + Apigee + Pub/Sub on GCP. Use when the user reports production incidents, latency spikes, partial outages, or asks to "see logs / metrics" in a GCP project.

- Skill: `aleonsa/gcp-debug` (Agent Skill)
- Install (CLI): `npx skillmds@latest add aleonsa/gcp-debug`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aleonsa/gcp-debug/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: aleonsa (https://skillmd.com/u/aleonsa)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/aleonsa/gcp-debug

---


# /gcp-debug — GCP Production Debugging Workflow

End-to-end diagnostic workflow for incidents on a Cloud Run + Apigee stack. Built from real incident triage (Intercom freeze caused by per-request `PublisherClient` instantiation + Apigee SpikeArrest matching the wrong path).

## When to activate

- User reports a prod incident: "the app is frozen", "we have latency", "5xx in service X"
- User asks to "look at logs", "check metrics", "see what happened at HH:MM"
- User says a third-party (webhook source like Intercom/Slack/WhatsApp) is throttling or freezing them
- Any time you need to correlate Apigee gateway behavior with backend Cloud Run behavior

## Core principles

- **Always confirm time window** in absolute UTC. Convert from user's local time (CDMX = UTC-6, Caracas = UTC-4) and state the conversion back to the user.
- **Triangulate, don't trust one signal.** A single dashboard can lie. Cross-check Cloud Run metrics against `run.googleapis.com/requests` log lines and against Apigee LB logs.
- **Look for "round" repeated values in latency** (e.g., 3.06s repeated across 8 requests = timeout signal, not random tail). This is the strongest fingerprint of a timeout/retry bug.
- **`p99 = 64.229s` in `request_latencies` means the histogram bucket is saturated** — actual values may be higher and Cloud Run is timing them out.
- **Never propose a fix until the chain of causation is complete.** Surface causes amplify each other; pick the one that, if removed, breaks the chain at its root.

## Gotchas you must know about this stack

| Gotcha | Workaround |
|---|---|
| `gcloud monitoring time-series` does NOT exist | Use REST API: `https://monitoring.googleapis.com/v3/projects/$PROJ/timeSeries?...` with `gcloud auth print-access-token` |
| Apigee Analytics add-on **disabled** in `cashea-apigee-prod` | The `/stats/apiproxy` API returns "Analytics add-on is disabled". Use Cloud Logging on `resource.type="http_load_balancer"` instead |
| `httpRequest.latency` is empty in app logs | Use `logName="projects/$PROJ/logs/run.googleapis.com%2Frequests"` to get real request logs with latency |
| Apigee `Matches "/foo/*"` matches **single segment**, not sub-paths | Use `MatchesPath "/foo/**"` for multi-segment paths |
| Cloud Run `instance_count` returns `state` label: `active` and `idle` | Sum them for total provisioned capacity. `idle` = warm but no traffic |
| If CPU < 60% target but instances pinned to `min`, the bottleneck is **I/O wait**, not CPU | Don't blame the autoscaler — look at LLM/DB/external HTTP latencies |

## Standard workflow

### Step 0 — Confirm scope

Ask the user (if not already given):
- Time window in absolute UTC
- Which service / channel / endpoint they suspect
- Whether there's a specific dashboard/screenshot they're looking at

### Step 1 — List what you have access to

```bash
gcloud config list                       # which account/project
gcloud projects list --limit=20          # what projects user has
gcloud run services list --project=$PROJ --format="table(metadata.name,status.url,...autoscaling...)"
```

### Step 2 — Get error logs in window

```bash
gcloud logging read 'resource.type="cloud_run_revision" AND severity>=ERROR \
  AND timestamp>="$START_UTC" AND timestamp<="$END_UTC"' \
  --project=$PROJ --limit=200 \
  --format="value(timestamp,resource.labels.service_name,severity,jsonPayload.message,textPayload)"
```

### Step 3 — Get traffic + latency metrics (REST, not gcloud)

```bash
TOKEN=$(gcloud auth print-access-token)
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://monitoring.googleapis.com/v3/projects/$PROJ/timeSeries?\
filter=metric.type%3D%22run.googleapis.com%2Frequest_latencies%22%20AND%20resource.label.service_name%3D%22$SVC%22&\
interval.startTime=$START_UTC&\
interval.endTime=$END_UTC&\
aggregation.alignmentPeriod=300s&\
aggregation.perSeriesAligner=ALIGN_PERCENTILE_99&\
aggregation.crossSeriesReducer=REDUCE_MAX"
```

Key metric types:
- `run.googleapis.com/request_latencies` — p50/p95/p99 (use `ALIGN_PERCENTILE_*`)
- `run.googleapis.com/request_count` — req/s (use `ALIGN_RATE`, group by `metric.label.response_code_class`)
- `run.googleapis.com/container/instance_count` — group by `metric.label.state` for active/idle
- `run.googleapis.com/container/cpu/utilizations` — to check if autoscaler should be triggering
- `run.googleapis.com/container/max_request_concurrencies` — to see saturation per instance

### Step 4 — Find slow individual requests

```bash
gcloud logging read 'logName="projects/$PROJ/logs/run.googleapis.com%2Frequests" \
  AND resource.labels.service_name="$SVC" \
  AND timestamp>="$START_UTC" AND timestamp<="$END_UTC"' \
  --project=$PROJ --limit=10000 \
  --format="value(timestamp,httpRequest.latency,httpRequest.status,httpRequest.requestUrl,httpRequest.userAgent)" \
  | awk -F'\t' '{l=$2; gsub("s","",l); print l+0"\t"$0}' \
  | sort -rn | head -30 | cut -f2-
```

This finds the actual slow requests. **Look for repeated near-identical latency values** — they signal a hard timeout or retry. Random tail ≠ repeated round number.

### Step 5 — Apigee LB logs (when proxy is suspected)

```bash
# Status code distribution for /your-proxy-path
gcloud logging read 'resource.type="http_load_balancer" \
  AND timestamp>="$START_UTC" AND timestamp<="$END_UTC" \
  AND httpRequest.requestUrl=~"/your-proxy-base"' \
  --project=$APIGEE_PROJ --limit=10000 \
  --format="value(httpRequest.status)" | sort | uniq -c | sort -rn

# Drill into specific status (e.g. 429)
gcloud logging read '... AND httpRequest.status=429' --format="value(timestamp,httpRequest.userAgent,httpRequest.requestUrl,httpRequest.latency)"
```

### Step 6 — Pull Apigee proxy bundle (config inspection)

```bash
TOKEN=$(gcloud auth print-access-token)
# Find active revision
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://apigee.googleapis.com/v1/organizations/$APIGEE_ORG/environments/$ENV/apis/$PROXY/deployments"

# Download bundle
curl -s -H "Authorization: Bearer $TOKEN" \
  -o /tmp/proxy.zip \
  "https://apigee.googleapis.com/v1/organizations/$APIGEE_ORG/apis/$PROXY/revisions/$REV?format=bundle"

unzip -o /tmp/proxy.zip -d /tmp/proxy-bundle
```

Then inspect XML files in `apiproxy/policies/` (look for `SpikeArrest`, `Quota`, `RateLimit`) and `apiproxy/proxies/default.xml` (look at `Condition` blocks — be wary of `Matches` vs `MatchesPath`).

### Step 7 — Build the chain of causation

Lay out a timeline like:

```
HH:MM:SS.ms  Event A (with source: log line / metric / proxy config)
HH:MM:SS.ms  Event B  (consequence of A)
HH:MM:SS.ms  Event C  (root cause - candidate fix)
```

Then identify which fix would break the chain at the highest leverage point. Surface causes (e.g., a 429 burst) often have a deeper cause (a 3s timeout that triggered the retry burst).

## Reporting format to user

Always present findings as:

1. **Hallazgo** — what you found (concrete numbers, log excerpts, file:line)
2. **Cadena de causación** — timeline + how each event leads to the next
3. **Fix(es) propuestos** — minimal, prioritized by impact, with diff
4. **Pregunta antes de actuar** — confirmation needed before applying anything

## Anti-patterns

- Running broad queries without time bounds — burns API quota and dumps too much
- Trusting a single percentile spike without correlating with other signals
- Proposing a fix at the symptom layer (e.g., raising SpikeArrest rate) without finding the root timeout/retry that caused the burst
- Skipping the Apigee bundle inspection when proxy 4xx/5xx is observed — the policy XML is often where the bug is
- Recommending autoscaler tuning when CPU is < 20% (the bottleneck is I/O wait, not capacity)

## Constraints

- Never modify GCP infra without explicit user approval (Apigee proxy changes especially — proxy is shared across teams)
- Never propose `--force`, `--no-verify`, or "raise the limit" type fixes as first resort
- Never skip the time-window confirmation step — wrong window invalidates the entire diagnosis
- Always state UTC + user's local time when reporting timestamps