GCP Debug

/gcp-debug — GCP Production Debugging Workflow

End-to-end diagnostic workflow for incidents on a Cloud Run + Apigee stack. Built from real incident triage (Intercom freeze caused by per-request PublisherClient instantiation + Apigee SpikeArrest matching the wrong path).

When to activate

• User reports a prod incident: "the app is frozen", "we have latency", "5xx in service X"
• User asks to "look at logs", "check metrics", "see what happened at HH:MM"
• User says a third-party (webhook source like Intercom/Slack/WhatsApp) is throttling or freezing them
• Any time you need to correlate Apigee gateway behavior with backend Cloud Run behavior

Core principles

• Always confirm time window in absolute UTC. Convert from user's local time (CDMX = UTC-6, Caracas = UTC-4) and state the conversion back to the user.
• Triangulate, don't trust one signal. A single dashboard can lie. Cross-check Cloud Run metrics against run.googleapis.com/requests log lines and against Apigee LB logs.
• Look for "round" repeated values in latency (e.g., 3.06s repeated across 8 requests = timeout signal, not random tail). This is the strongest fingerprint of a timeout/retry bug.
• p99 = 64.229s in request_latencies means the histogram bucket is saturated — actual values may be higher and Cloud Run is timing them out.
• Never propose a fix until the chain of causation is complete. Surface causes amplify each other; pick the one that, if removed, breaks the chain at its root.

Gotchas you must know about this stack

Standard workflow

Step 0 — Confirm scope

Ask the user (if not already given):

• Time window in absolute UTC
• Which service / channel / endpoint they suspect
• Whether there's a specific dashboard/screenshot they're looking at

Step 1 — List what you have access to

gcloud config list                       # which account/project
gcloud projects list --limit=20          # what projects user has
gcloud run services list --project=$PROJ --format="table(metadata.name,status.url,...autoscaling...)"

Step 2 — Get error logs in window

gcloud logging read 'resource.type="cloud_run_revision" AND severity>=ERROR \
  AND timestamp>="$START_UTC" AND timestamp<="$END_UTC"' \
  --project=$PROJ --limit=200 \
  --format="value(timestamp,resource.labels.service_name,severity,jsonPayload.message,textPayload)"

Step 3 — Get traffic + latency metrics (REST, not gcloud)

TOKEN=$(gcloud auth print-access-token)
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://monitoring.googleapis.com/v3/projects/$PROJ/timeSeries?\
filter=metric.type%3D%22run.googleapis.com%2Frequest_latencies%22%20AND%20resource.label.service_name%3D%22$SVC%22&\
interval.startTime=$START_UTC&\
interval.endTime=$END_UTC&\
aggregation.alignmentPeriod=300s&\
aggregation.perSeriesAligner=ALIGN_PERCENTILE_99&\
aggregation.crossSeriesReducer=REDUCE_MAX"

Key metric types:

• run.googleapis.com/request_latencies — p50/p95/p99 (use ALIGN_PERCENTILE_*)
• run.googleapis.com/request_count — req/s (use ALIGN_RATE, group by metric.label.response_code_class)
• run.googleapis.com/container/instance_count — group by metric.label.state for active/idle
• run.googleapis.com/container/cpu/utilizations — to check if autoscaler should be triggering
• run.googleapis.com/container/max_request_concurrencies — to see saturation per instance

Step 4 — Find slow individual requests

gcloud logging read 'logName="projects/$PROJ/logs/run.googleapis.com%2Frequests" \
  AND resource.labels.service_name="$SVC" \
  AND timestamp>="$START_UTC" AND timestamp<="$END_UTC"' \
  --project=$PROJ --limit=10000 \
  --format="value(timestamp,httpRequest.latency,httpRequest.status,httpRequest.requestUrl,httpRequest.userAgent)" \
  | awk -F'\t' '{l=$2; gsub("s","",l); print l+0"\t"$0}' \
  | sort -rn | head -30 | cut -f2-

This finds the actual slow requests. Look for repeated near-identical latency values — they signal a hard timeout or retry. Random tail ≠ repeated round number.

Step 5 — Apigee LB logs (when proxy is suspected)

# Status code distribution for /your-proxy-path
gcloud logging read 'resource.type="http_load_balancer" \
  AND timestamp>="$START_UTC" AND timestamp<="$END_UTC" \
  AND httpRequest.requestUrl=~"/your-proxy-base"' \
  --project=$APIGEE_PROJ --limit=10000 \
  --format="value(httpRequest.status)" | sort | uniq -c | sort -rn

# Drill into specific status (e.g. 429)
gcloud logging read '... AND httpRequest.status=429' --format="value(timestamp,httpRequest.userAgent,httpRequest.requestUrl,httpRequest.latency)"

Step 6 — Pull Apigee proxy bundle (config inspection)

TOKEN=$(gcloud auth print-access-token)
# Find active revision
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://apigee.googleapis.com/v1/organizations/$APIGEE_ORG/environments/$ENV/apis/$PROXY/deployments"

# Download bundle
curl -s -H "Authorization: Bearer $TOKEN" \
  -o /tmp/proxy.zip \
  "https://apigee.googleapis.com/v1/organizations/$APIGEE_ORG/apis/$PROXY/revisions/$REV?format=bundle"

unzip -o /tmp/proxy.zip -d /tmp/proxy-bundle

Then inspect XML files in apiproxy/policies/ (look for SpikeArrest, Quota, RateLimit) and apiproxy/proxies/default.xml (look at Condition blocks — be wary of Matches vs MatchesPath).

Step 7 — Build the chain of causation

Lay out a timeline like:

HH:MM:SS.ms  Event A (with source: log line / metric / proxy config)
HH:MM:SS.ms  Event B  (consequence of A)
HH:MM:SS.ms  Event C  (root cause - candidate fix)

Then identify which fix would break the chain at the highest leverage point. Surface causes (e.g., a 429 burst) often have a deeper cause (a 3s timeout that triggered the retry burst).

Reporting format to user

Always present findings as:

1. Hallazgo — what you found (concrete numbers, log excerpts, file:line)
2. Cadena de causación — timeline + how each event leads to the next
3. Fix(es) propuestos — minimal, prioritized by impact, with diff
4. Pregunta antes de actuar — confirmation needed before applying anything

Anti-patterns

• Running broad queries without time bounds — burns API quota and dumps too much
• Trusting a single percentile spike without correlating with other signals
• Proposing a fix at the symptom layer (e.g., raising SpikeArrest rate) without finding the root timeout/retry that caused the burst
• Skipping the Apigee bundle inspection when proxy 4xx/5xx is observed — the policy XML is often where the bug is
• Recommending autoscaler tuning when CPU is < 20% (the bottleneck is I/O wait, not capacity)

Constraints

• Never modify GCP infra without explicit user approval (Apigee proxy changes especially — proxy is shared across teams)
• Never propose --force, --no-verify, or "raise the limit" type fixes as first resort
• Never skip the time-window confirmation step — wrong window invalidates the entire diagnosis
• Always state UTC + user's local time when reporting timestamps