Agent Platform Troubleshooting
Important: You MUST ONLY use the reference files located in this skill's
references/ directory (references/field-manual.md,
references/known-issues.md, references/agent-gateway.md,
references/policies.md, references/agent-registry.md,
references/agent-identity.md). Do NOT search for or read external playbooks
outside this directory.
Diagnose issues across the Google Cloud Gemini Enterprise Agent Platform: Agent
Gateway, Agent Connectivity Templates (ACT), Agent Registry (Services, Agents,
MCP Servers, Endpoints), Agent Identity, IAP/UAP policies, Model Armor,
Semantic Governance Policies (SGP), explicit/transparent proxies, and tracing.
This skill produces a diagnostic report — findings and fix recommendations.
It does not apply fixes; the user owns all mutations.
Usage Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow Usage Attribution. Prefix
direct terminal gcloud calls (never generated scripts, Terraform, or user
code) with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:agent-platform-troubleshooting)"
and append agent-skills/1.0 (skill:agent-platform-troubleshooting) to direct
HTTP/REST/MCP User-Agent headers.
MANDATORY PRE-FLIGHT CHECKLIST (CHECK BEFORE RESPONDING OR CALLING TOOLS)
CRITICAL: Before generating ANY response or calling any tools, evaluate the
user's prompt against Rules 1–14. When a Turn-0 rule matches, execute its
directive immediately and STOP.
Rule 1: Out-of-Scope GCP IAM / GCS Queries
If the prompt asks about Compute Engine (GCE), Cloud Storage (GCS) buckets, or
generic GCP IAM permissions unrelated to the Agent Platform stack:
- IMMEDIATELY DECLINE ON TURN 0. DO NOT CALL ANY TOOLS OR GIVE IAM GUIDES.
- Respond: "I decline to troubleshoot generic GCP IAM or GCS access issues, as
they are out of scope for the Agent Platform Troubleshooting skill."
Rule 2: Strict Prohibition on Custom Discovery Scripts
If asked to write, generate, compile, or execute a custom Python or bash script
to discover resources (e.g., active Agent Runtime instances):
- DO NOT CALL ANY TOOLS OR WRITE/RUN ANY SCRIPTS.
- RESPOND ON TURN 0: "I cannot write or execute custom Python or bash
scripts for resource discovery. Custom discovery scripts are prohibited as
they consume excessive turns and cause timeouts. Instead, use standard gcloud
CLI commands (see Google Cloud SDK Installation)
or curl REST API calls with application default credentials:
gcloud agent-registry agents list --location=us-central1 or curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/reasoningEngines""
Rule 3: Consolidated Registry for Google APIs / Design Queries
If asked about registering multiple Agent Runtime, Cloud Resource Manager, or
Google API interfaces, or structuring services in Agent Registry:
- DO NOT CALL ANY TOOLS. RESPOND ON TURN 0 WITH:
- Consolidate ALL Google APIs under a single
googleapis service entry in
Agent Registry via gcloud agent-registry services create googleapis --endpoint-spec-type=no-spec.
- Explicitly state: "Do NOT register each Google API as a separate registry
service entry, as separate entries cause resource clutter, complicate IAM
policy management, and risk hitting registry quota limits."
- List all 8 required base FQDN interfaces (plus global
https://aiplatform.googleapis.com when calling global Gemini models):
https://agentregistry.googleapis.com
https://aiplatform.mtls.googleapis.com
https://cloudresourcemanager.mtls.googleapis.com
https://iamcredentials.mtls.googleapis.com
https://telemetry.mtls.googleapis.com
https://{region}-aiplatform.mtls.googleapis.com
https://{region}-aiplatform.googleapis.com
https://aiplatform.{region}.rep.googleapis.com
- Refer to
references/agent-registry.md §2 for the full command.
Rule 4: Cloud Run / Cloud Functions Egress 401/403 (OIDC vs IAP MCP)
If an agent gets a 401 or 403 egress error calling an MCP server or tool on
Cloud Run or Cloud Functions:
- DO NOT RUN LOG TOOLS. RESPOND ON TURN 0 WITH:
- Explain that direct Agent Identity (
principalSet://... / principal://...)
to standard Cloud Run OIDC authentication is not natively supported.
- Recommend Service Account impersonation in agent code to obtain an OIDC
ID token, granting
roles/iam.serviceAccountTokenCreator to the Agent
Identity on the target SA (and roles/run.invoker to that SA on Cloud Run).
- Note that for Preview IAP-enabled Cloud Run MCP servers (
--iap --functional-type=mcp-server, run.app short URL), authorize via IAP/UAP
instead of roles/run.invoker (see references/known-issues.md BKI 21 and
references/agent-identity.md).
Rule 5: Telemetry & Monitoring Endpoint Blocks on Startup
If Agent Runtime startup crashes (set_up()) with connection resets or
handshake errors reaching telemetry.mtls.googleapis.com:
- Evidence: Explicitly check whether all 4 monitoring/tracing endpoints are
registered as Endpoints in Agent Registry (
gcloud agent-registry endpoints list):
telemetry.mtls.googleapis.com, monitoring.googleapis.com,
trace.mtls.googleapis.com, and cloudtrace.googleapis.com.
- Recommended Fix:
- Register missing telemetry/tracing interfaces via
gcloud agent-registry services create --endpoint-spec-type=no-spec (which projects read-only
Endpoints in Agent Registry for telemetry.mtls.googleapis.com,
monitoring.googleapis.com, trace.mtls.googleapis.com, and
cloudtrace.googleapis.com).
- Create or update an
AuthorizationPolicy (AuthzPolicy / IAP / UAP)
bound to the Gateway that allows the agent's principal set (identity) to
access the telemetry endpoints (see references/known-issues.md BKI 23).
Rule 6: IAP Denial Troubleshooting (403 / Egress request is not authorized)
When logs or symptoms show 403 / Egress request is not authorized via IAP:
- Check IAP Egressor bindings on the registry entry FIRST (
roles/iap.egressor
via gcloud iap web get-iam-policy --resource-type=agent-registry bound to
the server-generated child UUID from registryResource, not SERVICE_NAME).
- Create a registry entry (
gcloud agent-registry services create) if none
exists for the exact target hostname.
- Grant
roles/iap.egressor on the registry entry (IAM v1 — set-iam-policy requires top-level {"bindings": [...]} JSON without a "policy" wrapper) or verify UAP
AccessPolicy + CRM PolicyBinding (iapPolicyVersion: "V2",
iap.googleapis.com/resources.egressViaIAP; see references/policies.md §2).
- Check IAP audit logs (
protoPayload.serviceName="iap.googleapis.com")
and verify an AuthorizationPolicy (AuthzPolicy) is bound to the Gateway.
- Explicitly warn: "Do NOT use
roles/iap.tunnelResourceAccessor (which
is for classic IAP TCP tunneling to GCE VMs, not Agent Gateway egress) and do
NOT bypass IAP authentication."
Rule 7: PSC Subnet Exhaustion Speed Rule (PROVISIONING Hang)
When diagnosing gateway provisioning stuck or failing on subnet IP exhaustion:
- DO NOT loop across regions. Run ONLY in the target region (default
us-central1):
gcloud network-services agent-gateways list --location=us-central1
gcloud network-services agent-gateways describe <GW> --location=us-central1
gcloud compute network-attachments describe <NA> --region=us-central1
gcloud compute networks subnets describe <SUBNET> --region=us-central1
- Check free IP count (
Usable - Allocated = Free) before recommending any
Network Attachment deletion; flag that a /28 subnet is easily exhausted and
recommend expanding the CIDR range to at least /26 (known-issues.md BKI 19).
Rule 8: Multi-Region Manual Registration Prohibition (us / eu)
If asked about manually registering endpoints or services in us or eu:
- DO NOT CALL ANY TOOLS. RESPOND ON TURN 0:
- "Manual endpoint/service registration is NOT supported in
us or eu
multi-region locations (us and eu are read-only roll-up locations for
auto-discovered resources)." (Explicitly name BOTH us and eu.)
- "Register manually in a specific region (e.g.,
us-central1) or global."
Rule 9: VPC Service Controls (VPC-SC) Perimeter & ACT ALL_TRAFFIC
When diagnosing VPC-SC perimeter blocks or external API hangs under VPC-SC:
- Identify a VPC Service Controls perimeter block / boundary enforcement.
- Check that the Agent Connectivity Template (ACT) sets
vpcEgress: ALL_TRAFFIC (native VPC-SC creation works out of the box since Sept 8,
2026 when vpcEgress: ALL_TRAFFIC is set) and that Cloud NAT is
configured on the consumer VPC's PSC-I network attachment subnet.
- For legacy or strict custom perimeters blocking Google actuation identities,
recommend creating ingress policies allowing:
actuation-a@networkservices-prod.iam.gserviceaccount.com
cloud-aiplatform-pipeline-robot-prod@system.gserviceaccount.com
- Explicitly state: "Do NOT disable VPC Service Controls or delete perimeters."
Refer to
references/known-issues.md BKI 18 & 28.
Rule 10: UAP Policy Binding Org Policy Blocker & Blanket DENY Crash
CUSTOM_ORG_POLICY_VIOLATION on gcloud iam policy-bindings create:
Caused by constraints/iam.managed.disableAccessPolicyBinding (note: some
docs misspell this plural). Apply an Org Policy override (enforce: false) at
the target resource and wait 30 to 60 seconds for IAM propagation (BKI 24).
- Startup crash after UAP
destination.is_registered == false DENY:
Explain that the blanket DENY blocked internal/unauthenticated SDK bootstrap
and metadata calls (such as Cloud Resource Manager project lookups) during
Reasoning Engine init; remove or scope the blanket DENY rule (BKI 5).
Rule 11: Agent Gateway Dual-Registry & Jurisdiction Invariants
When configuring or validating registries on an Agent Gateway:
- An Agent Gateway supports a maximum of two registries.
- When two registries are configured, exactly one must be
global and the
other must be regional or multi-regional.
- Binding two regional registries (e.g.,
us-central1 + europe-west1), two
multi-regionals, two globals, or a mismatched jurisdiction (e.g., us-central1
gateway + eu registry) is invalid and rejected with an HTTP 400
validation error (references/agent-gateway.md §4, BKI 26).
Rule 12: Cross-Project Runtime-to-Gateway Binding Diagnosis
When Reasoning Engine in Project 1 fails to bind to a Gateway in Project 2:
- Deploying Identity: Needs
networkservices.agentGateways.use and
networkservices.agentGateways.get (or roles/networkservices.viewer) in Project 2.
- Vertex AI Service Agent: Grant
roles/networkservices.viewer or custom
role ae_agw_cross_project_sa (alias ar_agw_cross_project_sa) to
service-PROJECT_1_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com in Project 2.
- Regional Colocation: Verify both runtime and gateway are in the same region.
- CAA Opt-Out: Set
GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES
to False (references/known-issues.md BKI 10, references/agent-gateway.md §11).
Rule 13: Downstream SWP + PBR vs. Explicit Forward HTTPS Proxy
Distinguish the two supported consumer-VPC proxy architectures:
- Pattern A — Transparent Next-Hop SWP + PBR (no client
HTTPS_PROXY needed):
- Explain that PSC endpoints cannot be route next-hops, SWP behind PSC
Service Attachments only supports explicit proxy mode (not transparent
next-hop mode), and static routes cannot be used because Network
Attachments cannot have VM network tags.
- Deploy SWP directly in the consumer VPC with
type: SECURE_WEB_GATEWAY and
routingMode: NEXT_HOP_ROUTING_MODE on an internal IP on the PSC-I subnet
(e.g., 10.20.1.250), with a /26 REGIONAL_MANAGED_PROXY ACTIVE subnet.
- Configure Policy-Based Routing (PBR) matching source CIDR
10.20.1.0/24
to --next-hop-ilb-ip=10.20.1.250 (priority 200) + fallback PBR (1000).
- Create Cloud NAT with
ENDPOINT_TYPE_SWG (nats create ... --endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG),
and allowlist <region>-aiplatform.mtls.googleapis.com in SWP CEL (BKI 33).
- Pattern B — Explicit Forward HTTPS Proxy (
HTTPS_PROXY + L4 AuthzPolicy):
- Without L4 bypass,
HTTPS_PROXY=https://proxy.corp.internal:443 fails on
HTTP CONNECT with HTTP 404 because Agent Gateway's L7 HTTP manager
terminates TLS and rejects nested CONNECT.
- Import a
v1beta1 L4 AuthzPolicy via gcloud beta network-security authz-policies import with action: ALLOW and networkRules matching
to.operations[].snis (omit target.loadBalancingScheme and ports; max 5 rules;
omit from or include 7.0.0.0/8 + PSC-I CIDR in from.sources[].ipBlocks).
- Listen with TLS on
:443, allow the consumer PSC-I subnet CIDR on VPC firewall/proxy ACLs (never 7.0.0.0/8),
set NO_PROXY/no_proxy to .googleapis.com,googleapis.com,metadata.google.internal,169.254.169.254,localhost,127.0.0.1,
bundle agentGatewayCard.rootCertificates + Private CA in container env
vars (SSL_CERT_FILE, REQUESTS_CA_BUNDLE), and warn L4 ALLOW bypasses
L7 IAP/UAP, Model Armor, SGP, and httpRules for matched SNIs (BKI 34).
Rule 14: DNS Peering, TrustConfig, Model Armor, SGP, MCP, Identity, & Lifecycle
- DNS Peering & Service Agent: Public FQDNs resolve via public recursive DNS.
Private DNS requires
dnsPeeringConfig (domains array with trailing .; ACT
uses string targetNetwork only, whereas inline AgentGateway.networkConfig.dnsPeeringConfig
also requires targetProject). Cloud DNS caps total peering hops at 2
(Gateway consumes Hop 1 → max 1 consumer peering hop; a 2nd hop fails preflight
or returns SERVFAIL → 503, BKI 35). Grant roles/dns.peer, roles/dns.reader,
roles/compute.networkViewer, and roles/compute.networkUser to
service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com
(and compute.networkAttachments.use to caller, BKI 9). Never use authoritative "." private root zones ("." requires a Cloud DNS Private Forwarding Zone).
- Private CA
TrustConfig & Immutable ACT: Set egressNetworkConfig.tlsConfig.trustConfig
with additionalRoots: PUBLICLY_TRUSTED_ROOTS (omitting breaks *.googleapis.com).
ACT fields are immutable — create a new ACT and rebind via PATCH with
updateMask=agentConnectivityTemplate (BKI 6).
- Model Armor (
CONTENT_AUTHZ): Ingress (CLIENT_TO_AGENT, max 1, ADK
streamQuery only) uses service-AGENT_PROJECT_NUMBER@gcp-sa-aiplatform-re.iam.gserviceaccount.com;
Egress (AGENT_TO_ANYWHERE, max 4) uses service-GATEWAY_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com.
Scope httpRules when to exclude gRPC and streams >60 seconds (Stream removed, BKI 13).
- Semantic Governance Policies (SGP): Uses
gcp-sa-aiplatform, non-streaming
LLM calls only, .contains() + !request.headers['content-type'].startsWith('application/grpc')
in when, and full-map updates on PATCH ?updateMask=metadata so
sgpEnforcementMode: "DRY_RUN" and sgpUserIntentMatching are preserved (BKI 38).
- Native MCP, Identity, Trace, & Teardown: Set
baseProtocolMethodsOption: MATCH_BASE_PROTOCOL_METHODS on MCP ALLOW rules (BKI 39). Orgless trust
domains use agents.global.proj-PROJECT_NUMBER.system.id.goog (proj-, never
project-); never grant roles/aiplatform.expressUser or project-wide
roles/aiplatform.user to Agent Identities. For Cloud Trace (TelemetryPolicy),
always recommend enabling cloudtrace.googleapis.com (gcloud services enable cloudtrace.googleapis.com)
and granting roles/cloudtrace.agent to compute-system, gcp-sa-networksecurity,
and gcp-sa-agentgateway (see references/agent-gateway.md §8.3 and references/field-manual.md Step 14.4). Tear down in strict 7-step order
(workloads → UAP bindings → AuthzPolicy → extensions → AgentGateway → ACT →
VPC) and use unique gateway IDs if deletion tombstones collide (BKIs 25, 30).
Quick Symptom Triage Matrix
| Symptom / Error |
Primary Suspect |
Reference |
403 / Egress request is not authorized |
Unregistered FQDN, missing roles/iap.egressor / UAP binding, or PAB |
Rules 4, 6; references/policies.md §2; BKIs 4, 7, 8, 15 |
HTTP 404 on HTTP CONNECT (HTTPS_PROXY) |
Missing gcloud beta L4 AuthzPolicy (networkRules snis) |
Rule 13B; references/agent-gateway.md §6.3; BKI 34 |
HTTP 498 immediately after binding runtime |
Enforcing gateway bound before allowlisting essential platform APIs |
references/agent-gateway.md §2; BKI 37 |
500 / PERMISSION_DENIED / Stream removed |
Model Armor SA roles (gcp-sa-dep vs gcp-sa-aiplatform-re) or 60s stream cap |
Rule 14; references/policies.md §12; BKI 13 |
503 / NXDOMAIN / SERVFAIL on private host |
>2-hop DNS peering, authoritative "." zone, Packet Intercept, or SWP |
Rules 13A, 14; references/agent-gateway.md §3, §6; BKIs 33, 35, 36 |
Config validation failed / PERMISSION_DENIED on import |
Disabled agentregistry.googleapis.com, missing caller/gcp-sa-agentgateway IAM, or bad NA URI |
Rule 14; BKIs 9, 20, 26 |
HTTP 409 ABORTED / 400 already being used |
Active control-plane LRO mutex (~2 min), tombstone collision, or reverse teardown |
references/agent-gateway.md §9, §10; BKIs 25, 29, 30 |
Startup crash (reasoning_engine_stderr) |
Blocked telemetry/CRM bootstrap, UAP blanket DENY, or SGP gRPC interception |
Rules 5, 10, 14; BKIs 1, 5, 17, 23, 38 |
When to use this skill
Trigger on Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry,
Agent Identity, IAP/UAP policies, Model Armor, Semantic Governance Policies
(SGP), SWP/Explicit Proxy egress, Private CA TrustConfig, Cloud DNS peering,
Cloud Trace TelemetryPolicy, or Gemini Enterprise / Agent Runtime / GKE /
Cloud Run gateway bindings. Do not use for generic Google Cloud IAM, GCS,
GCE, or networking issues unrelated to the Agent Platform stack.
Required context & Hypothesis Rules
Gather PROJECT_ID, PROJECT_NUMBER, LOCATION (default us-central1),
AGENT_ID, AGENT_GATEWAY_NAME, Agent Identity (principal:// or
principalSet://), exact error + timestamp, and target destination. If missing,
ask or state unknowns using placeholders — never scan across projects or loop
over regions. Before Step 1 queries, formulate at most 3 hypotheses anchored
to "Why did it start failing now?" (e.g., recent Terraform/config changes).
Diagnostic flow
Follow in order (copy-paste commands in references/field-manual.md):
- Step 0 — Pre-Flight & Access: Evaluate Rules 1–14. Verify project access
(
gcloud projects describe $PROJECT_ID).
- Step 1 — Agent & Startup Logs: Classify error (
403, 404 CONNECT, 498,
500, 503, crash, timeout). Check PSC subnet IPs, ACT ALL_TRAFFIC + Cloud
NAT, and agentGatewayCard.rootCertificates CA trust.
- Step 2 — Gateway Logs: Query
networkservices.googleapis.com/Gateway
(filtering -httpRequest.requestMethod="CONNECT" when inspecting L7 requests)
to find the exact failing SNI/hostname and serviceExtensionsInfo.
- Step 3 — IAP / Model Armor / SGP Logs: Check
iapPolicyVersion (V1 vs
V2), DRY_RUN vs enforced, Model Armor callout status, or
logs/semantic-governance-policy verdicts and rationales.
- Step 4 — Registry State: Verify exact hostname permutations in
gcloud agent-registry {services,endpoints,mcp-servers,agents} list and
confirm dual-registry + jurisdiction rules.
- Step 5 — Identity & Policies: Check
roles/iap.egressor via
--resource-type=agent-registry on the child UUID (V1), UAP
AccessPolicy/PolicyBinding + constraints/iam.managed.disableAccessPolicyBinding
(V2), PAB overrides, and least-privilege Agent Identity roles (proj- for orgless).
- Step 6 — AuthzPolicy & Extensions: Verify L4
networkRules vs L7
httpRules, MCP MATCH_BASE_PROTOCOL_METHODS, Model Armor / SGP when
exclusions, ExtensionBinding scope, and policy caps (max 4 egress, max 1 ingress CONTENT_AUTHZ).
- Step 7 — Connectivity, DNS, & TLS: Verify ACT
TrustConfig
(PUBLICLY_TRUSTED_ROOTS), gcp-sa-agentgateway IAM, <=2 DNS peering hops,
no Packet Intercept on the consumer VPC, and SWP/Explicit Proxy settings.
- Step 8 — PrincipalSet & Trace Verification: Test 1:1
principal://
binding if principalSet:// lags; for Cloud Trace, enable cloudtrace.googleapis.com, verify v1beta1 TelemetryPolicy (telemetryTarget referencing agentGateways/* and object parentBasedSampling: {enabled: true, samplingRate: 1.0}, never string ENABLED), and grant roles/cloudtrace.agent to service-<PROJECT_NUMBER>@compute-system.iam.gserviceaccount.com, service-<PROJECT_NUMBER>@gcp-sa-networksecurity.iam.gserviceaccount.com, and service-<PROJECT_NUMBER>@gcp-sa-agentgateway.iam.gserviceaccount.com.
Tools & Layered References
Use mcp__gcloud__run_gcloud_command (or gcloud --quiet),
mcp__gcloud-observability__list_log_entries (or gcloud logging read),
mcp__google-dev-knowledge__search_documents / get_documents / answer_query,
and curl REST GETs, plus bundled references (load only the smallest set needed):
- references/field-manual.md — read first for operational commands & log queries
- references/known-issues.md — BKIs #1–#13, #15–#39 for fast pattern matches
- references/agent-gateway.md — topologies, ACT, DNS, TLS, SWP/Explicit Proxy, Trace, teardown
- references/policies.md — L4/L7
AuthzPolicy, MCP, UAP v3beta, Model Armor, SGP
- references/agent-registry.md —
services vs projected endpoints, 9 API families, jurisdictions
- references/agent-identity.md — 3-identity delegation, Cloud Run OIDC vs IAP MCP, WIF
Output report
Always produce a read-only diagnostic report using this exact structure:
# Agent Platform Diagnostic — <one-line summary>
## Context
- Project: <id> (<number>) | Location: <region>
- Agent: <agent_id> | Gateway: <gateway_name>
- Symptom: <exact error message and when it started>
## Evidence gathered
- Agent log query: <filter, summary>
- Gateway log query: <filter, exact failing hostname / extension status>
- IAP / Policy log query: <filter, decision + enforcement mode>
- Registry & Policy state: <entries, AuthzPolicy bindings, roles/iap.egressor or UAP>
## Root cause hypothesis
<ranked root cause(s)>
## Why this fits the evidence
<connect log/config evidence to root cause>
## Recommended fix
<ordered gcloud / curl / YAML / Terraform changes for the user to apply>
## What to verify after the fix
<verification queries>
## Open questions / unknowns
<missing context or unverified items>
## Appendix: Raw Logs & Verified Links
- **Cloud Logging Filter**: <exact copy-pasteable filter or deep link>
- **Raw Logs**: <untruncated Agent, Gateway, and IAP/Policy log entries>
Principles
- Exact hostname match first: Compare gateway log hostnames directly against Agent Registry.
- Layered default-deny: Registry → Gateway
AuthzPolicy → Extension → IAP/UAP/Model Armor/SGP → PAB (PAB overrides IAM Allow).
- L4 bypass vs L7 governance: L4
networkRules enables HTTP CONNECT explicit proxies (:443) but bypasses L7 IAP, Model Armor, and SGP for matched SNIs.
- Read-only, single-region, non-interactive: Pull evidence first, never mutate resources, check
us-central1 unless logs indicate otherwise, and pass --quiet.
Supporting Links
1---2name: agent-platform-troubleshooting3description: >-4 Troubleshoots Google Cloud Gemini Enterprise Agent Platform issues across Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry, Agent Identity, Identity-Aware Proxy (IAP), Unified Access Policies (UAP), Model Armor, Semantic Governance Policies (SGP), Cloud DNS peering, Private CA TrustConfig, Next-Hop Secure Web Proxy (SWP), Explicit HTTPS Proxy (L4 AuthzPolicy), and Cloud Trace.5 Use when agent requests fail with 401/403/404/498/500/503 or Code 13, Egress request is not authorized, HTTP CONNECT fails with 404, Agent Runtime crashes on startup, or gateway/ACT/IAP/Model Armor/SGP configs fail validation.6 Don't use for general Google Cloud Identity and Access Management (IAM) debugging or networking issues unrelated to the Agent Platform stack.7metadata:8 source: https://skillmd.com/skills/google/agent-platform-troubleshooting9 version: "1.1.1"10 category: AiAndMachineLearning11---1213# Agent Platform Troubleshooting1415> **Important:** You MUST ONLY use the reference files located in this skill's16> `references/` directory (`references/field-manual.md`,17> `references/known-issues.md`, `references/agent-gateway.md`,18> `references/policies.md`, `references/agent-registry.md`,19> `references/agent-identity.md`). Do NOT search for or read external playbooks20> outside this directory.2122Diagnose issues across the Google Cloud Gemini Enterprise Agent Platform: Agent23Gateway, Agent Connectivity Templates (ACT), Agent Registry (Services, Agents,24MCP Servers, Endpoints), Agent Identity, IAP/UAP policies, Model Armor,25Semantic Governance Policies (SGP), explicit/transparent proxies, and tracing.26This skill produces a **diagnostic report** — findings and fix recommendations.27It does not apply fixes; the user owns all mutations.2829## Usage Attribution3031**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you32MUST read and follow [Usage Attribution](references/attribution.md). Prefix33direct terminal `gcloud` calls (never generated scripts, Terraform, or user34code) with35`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:agent-platform-troubleshooting)"`36and append `agent-skills/1.0 (skill:agent-platform-troubleshooting)` to direct37HTTP/REST/MCP `User-Agent` headers.3839## MANDATORY PRE-FLIGHT CHECKLIST (CHECK BEFORE RESPONDING OR CALLING TOOLS)4041CRITICAL: Before generating ANY response or calling any tools, evaluate the42user's prompt against Rules 1–14. When a Turn-0 rule matches, execute its43directive immediately and STOP.4445### Rule 1: Out-of-Scope GCP IAM / GCS Queries46If the prompt asks about Compute Engine (GCE), Cloud Storage (GCS) buckets, or47generic GCP IAM permissions unrelated to the Agent Platform stack:48- **IMMEDIATELY DECLINE ON TURN 0. DO NOT CALL ANY TOOLS OR GIVE IAM GUIDES.**49- Respond: *"I decline to troubleshoot generic GCP IAM or GCS access issues, as50 they are out of scope for the Agent Platform Troubleshooting skill."*5152### Rule 2: Strict Prohibition on Custom Discovery Scripts53If asked to write, generate, compile, or execute a custom Python or bash script54to discover resources (e.g., active Agent Runtime instances):55- **DO NOT CALL ANY TOOLS OR WRITE/RUN ANY SCRIPTS.**56- **RESPOND ON TURN 0**: *"I cannot write or execute custom Python or bash57 scripts for resource discovery. Custom discovery scripts are prohibited as58 they consume excessive turns and cause timeouts. Instead, use standard gcloud59 CLI commands (see [Google Cloud SDK Installation](https://docs.cloud.google.com/sdk/docs/install-sdk.md.txt))60 or curl REST API calls with application default credentials:61 `gcloud agent-registry agents list --location=us-central1` or `curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/reasoningEngines"`"*6263### Rule 3: Consolidated Registry for Google APIs / Design Queries64If asked about registering multiple Agent Runtime, Cloud Resource Manager, or65Google API interfaces, or structuring services in Agent Registry:66- **DO NOT CALL ANY TOOLS. RESPOND ON TURN 0 WITH**:67 1. Consolidate ALL Google APIs under a single `googleapis` service entry in68 Agent Registry via `gcloud agent-registry services create googleapis --endpoint-spec-type=no-spec`.69 2. Explicitly state: *"Do NOT register each Google API as a separate registry70 service entry, as separate entries cause resource clutter, complicate IAM71 policy management, and risk hitting registry quota limits."*72 3. List all 8 required base FQDN interfaces (plus global73 `https://aiplatform.googleapis.com` when calling global Gemini models):74 - `https://agentregistry.googleapis.com`75 - `https://aiplatform.mtls.googleapis.com`76 - `https://cloudresourcemanager.mtls.googleapis.com`77 - `https://iamcredentials.mtls.googleapis.com`78 - `https://telemetry.mtls.googleapis.com`79 - `https://{region}-aiplatform.mtls.googleapis.com`80 - `https://{region}-aiplatform.googleapis.com`81 - `https://aiplatform.{region}.rep.googleapis.com`82 4. Refer to `references/agent-registry.md` §2 for the full command.8384### Rule 4: Cloud Run / Cloud Functions Egress 401/403 (OIDC vs IAP MCP)85If an agent gets a `401` or `403` egress error calling an MCP server or tool on86Cloud Run or Cloud Functions:87- **DO NOT RUN LOG TOOLS. RESPOND ON TURN 0 WITH**:88 1. Explain that direct Agent Identity (`principalSet://...` / `principal://...`)89 to standard Cloud Run OIDC authentication is **not natively supported**.90 2. Recommend **Service Account impersonation** in agent code to obtain an OIDC91 ID token, granting **`roles/iam.serviceAccountTokenCreator`** to the Agent92 Identity on the target SA (and `roles/run.invoker` to that SA on Cloud Run).93 3. Note that for Preview IAP-enabled Cloud Run MCP servers (`--iap94 --functional-type=mcp-server`, `run.app` short URL), authorize via IAP/UAP95 instead of `roles/run.invoker` (see `references/known-issues.md` BKI 21 and96 `references/agent-identity.md`).9798### Rule 5: Telemetry & Monitoring Endpoint Blocks on Startup99If Agent Runtime startup crashes (`set_up()`) with connection resets or100handshake errors reaching `telemetry.mtls.googleapis.com`:101- **Evidence**: Explicitly check whether all 4 monitoring/tracing endpoints are102 registered as Endpoints in Agent Registry (`gcloud agent-registry endpoints list`):103 `telemetry.mtls.googleapis.com`, `monitoring.googleapis.com`,104 `trace.mtls.googleapis.com`, and `cloudtrace.googleapis.com`.105- **Recommended Fix**:106 1. Register missing telemetry/tracing interfaces via `gcloud agent-registry107 services create --endpoint-spec-type=no-spec` (which projects read-only108 Endpoints in Agent Registry for `telemetry.mtls.googleapis.com`,109 `monitoring.googleapis.com`, `trace.mtls.googleapis.com`, and110 `cloudtrace.googleapis.com`).111 2. Create or update an **`AuthorizationPolicy`** (`AuthzPolicy` / IAP / UAP)112 bound to the Gateway that allows the agent's principal set (identity) to113 access the telemetry endpoints (see `references/known-issues.md` BKI 23).114115### Rule 6: IAP Denial Troubleshooting (`403` / `Egress request is not authorized`)116When logs or symptoms show `403` / `Egress request is not authorized` via IAP:1171. **Check IAP Egressor bindings on the registry entry FIRST** (`roles/iap.egressor`118 via `gcloud iap web get-iam-policy --resource-type=agent-registry` bound to119 the server-generated child UUID from `registryResource`, not `SERVICE_NAME`).1202. **Create a registry entry** (`gcloud agent-registry services create`) if none121 exists for the exact target hostname.1223. **Grant `roles/iap.egressor`** on the registry entry (IAM v1 — `set-iam-policy` requires top-level `{"bindings": [...]}` JSON without a `"policy"` wrapper) or verify UAP123 `AccessPolicy` + CRM `PolicyBinding` (`iapPolicyVersion: "V2"`,124 `iap.googleapis.com/resources.egressViaIAP`; see `references/policies.md` §2).1254. **Check IAP audit logs** (`protoPayload.serviceName="iap.googleapis.com"`)126 and verify an `AuthorizationPolicy` (`AuthzPolicy`) is bound to the Gateway.1275. **Explicitly warn**: *"Do NOT use `roles/iap.tunnelResourceAccessor` (which128 is for classic IAP TCP tunneling to GCE VMs, not Agent Gateway egress) and do129 NOT bypass IAP authentication."*130131### Rule 7: PSC Subnet Exhaustion Speed Rule (`PROVISIONING` Hang)132When diagnosing gateway provisioning stuck or failing on subnet IP exhaustion:133- **DO NOT loop across regions.** Run ONLY in the target region (default `us-central1`):134 1. `gcloud network-services agent-gateways list --location=us-central1`135 2. `gcloud network-services agent-gateways describe <GW> --location=us-central1`136 3. `gcloud compute network-attachments describe <NA> --region=us-central1`137 4. `gcloud compute networks subnets describe <SUBNET> --region=us-central1`138- Check free IP count (`Usable - Allocated = Free`) **before** recommending any139 Network Attachment deletion; flag that a `/28` subnet is easily exhausted and140 recommend expanding the CIDR range to at least `/26` (`known-issues.md` BKI 19).141142### Rule 8: Multi-Region Manual Registration Prohibition (`us` / `eu`)143If asked about manually registering endpoints or services in `us` or `eu`:144- **DO NOT CALL ANY TOOLS. RESPOND ON TURN 0**:145 1. *"Manual endpoint/service registration is NOT supported in `us` or `eu`146 multi-region locations (`us` and `eu` are read-only roll-up locations for147 auto-discovered resources)."* (Explicitly name BOTH `us` and `eu`.)148 2. *"Register manually in a specific region (e.g., `us-central1`) or `global`."*149150### Rule 9: VPC Service Controls (VPC-SC) Perimeter & ACT `ALL_TRAFFIC`151When diagnosing VPC-SC perimeter blocks or external API hangs under VPC-SC:1521. Identify a **VPC Service Controls perimeter block** / boundary enforcement.1532. Check that the Agent Connectivity Template (ACT) sets **`vpcEgress:154 ALL_TRAFFIC`** (native VPC-SC creation works out of the box since Sept 8,155 2026 when `vpcEgress: ALL_TRAFFIC` is set) and that **Cloud NAT** is156 configured on the consumer VPC's PSC-I network attachment subnet.1573. For legacy or strict custom perimeters blocking Google actuation identities,158 recommend creating **ingress policies** allowing:159 - `actuation-a@networkservices-prod.iam.gserviceaccount.com`160 - `cloud-aiplatform-pipeline-robot-prod@system.gserviceaccount.com`1614. Explicitly state: *"Do NOT disable VPC Service Controls or delete perimeters."*162 Refer to `references/known-issues.md` BKI 18 & 28.163164### Rule 10: UAP Policy Binding Org Policy Blocker & Blanket `DENY` Crash165- **`CUSTOM_ORG_POLICY_VIOLATION` on `gcloud iam policy-bindings create`**:166 Caused by **`constraints/iam.managed.disableAccessPolicyBinding`** (note: some167 docs misspell this plural). Apply an Org Policy override (`enforce: false`) at168 the target resource and wait **30 to 60 seconds** for IAM propagation (BKI 24).169- **Startup crash after UAP `destination.is_registered == false` `DENY`**:170 Explain that the blanket `DENY` blocked internal/unauthenticated SDK bootstrap171 and metadata calls (such as Cloud Resource Manager project lookups) during172 Reasoning Engine init; remove or scope the blanket `DENY` rule (BKI 5).173174### Rule 11: Agent Gateway Dual-Registry & Jurisdiction Invariants175When configuring or validating `registries` on an Agent Gateway:1761. An Agent Gateway supports a **maximum of two registries**.1772. When two registries are configured, **exactly one must be `global`** and the178 other must be **`regional` or `multi-regional`**.1793. Binding two regional registries (e.g., `us-central1` + `europe-west1`), two180 multi-regionals, two globals, or a mismatched jurisdiction (e.g., `us-central1`181 gateway + `eu` registry) is invalid and rejected with an **HTTP 400**182 validation error (`references/agent-gateway.md` §4, BKI 26).183184### Rule 12: Cross-Project Runtime-to-Gateway Binding Diagnosis185When Reasoning Engine in Project 1 fails to bind to a Gateway in Project 2:1861. **Deploying Identity**: Needs `networkservices.agentGateways.use` and187 `networkservices.agentGateways.get` (or `roles/networkservices.viewer`) in Project 2.1882. **Vertex AI Service Agent**: Grant `roles/networkservices.viewer` or custom189 role `ae_agw_cross_project_sa` (alias `ar_agw_cross_project_sa`) to190 `service-PROJECT_1_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com` in Project 2.1913. **Regional Colocation**: Verify both runtime and gateway are in the **same region**.1924. **CAA Opt-Out**: Set `GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES`193 to `False` (`references/known-issues.md` BKI 10, `references/agent-gateway.md` §11).194195### Rule 13: Downstream SWP + PBR vs. Explicit Forward HTTPS Proxy196Distinguish the two supported consumer-VPC proxy architectures:197- **Pattern A — Transparent Next-Hop SWP + PBR** (no client `HTTPS_PROXY` needed):198 1. Explain that **PSC endpoints cannot be route next-hops**, **SWP behind PSC199 Service Attachments only supports explicit proxy mode (not transparent200 next-hop mode)**, and **static routes cannot be used because Network201 Attachments cannot have VM network tags**.202 2. Deploy SWP directly in the consumer VPC with `type: SECURE_WEB_GATEWAY` and203 `routingMode: NEXT_HOP_ROUTING_MODE` on an internal IP on the PSC-I subnet204 (e.g., `10.20.1.250`), with a `/26` `REGIONAL_MANAGED_PROXY` `ACTIVE` subnet.205 3. Configure **Policy-Based Routing (PBR)** matching source CIDR `10.20.1.0/24`206 to `--next-hop-ilb-ip=10.20.1.250` (priority 200) + fallback PBR (1000).207 4. Create **Cloud NAT with `ENDPOINT_TYPE_SWG`** (`nats create ... --endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG`),208 and allowlist `<region>-aiplatform.mtls.googleapis.com` in SWP CEL (BKI 33).209- **Pattern B — Explicit Forward HTTPS Proxy (`HTTPS_PROXY` + L4 `AuthzPolicy`)**:210 1. Without L4 bypass, `HTTPS_PROXY=https://proxy.corp.internal:443` fails on211 `HTTP CONNECT` with **HTTP `404`** because Agent Gateway's L7 HTTP manager212 terminates TLS and rejects nested `CONNECT`.213 2. Import a **`v1beta1`** L4 `AuthzPolicy` via **`gcloud beta network-security214 authz-policies import`** with `action: ALLOW` and `networkRules` matching215 `to.operations[].snis` (omit `target.loadBalancingScheme` and `ports`; max 5 rules;216 omit `from` or include `7.0.0.0/8` + PSC-I CIDR in `from.sources[].ipBlocks`).217 3. Listen with TLS on `:443`, allow the consumer PSC-I subnet CIDR on VPC firewall/proxy ACLs (never `7.0.0.0/8`),218 set `NO_PROXY`/`no_proxy` to `.googleapis.com,googleapis.com,metadata.google.internal,169.254.169.254,localhost,127.0.0.1`,219 bundle `agentGatewayCard.rootCertificates` + Private CA in container env220 vars (`SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`), and warn L4 `ALLOW` bypasses221 L7 IAP/UAP, Model Armor, SGP, and `httpRules` for matched SNIs (BKI 34).222223### Rule 14: DNS Peering, `TrustConfig`, Model Armor, SGP, MCP, Identity, & Lifecycle224- **DNS Peering & Service Agent**: Public FQDNs resolve via public recursive DNS.225 Private DNS requires `dnsPeeringConfig` (`domains` array with trailing `.`; ACT226 uses string `targetNetwork` only, whereas inline `AgentGateway.networkConfig.dnsPeeringConfig`227 also requires `targetProject`). Cloud DNS caps total peering hops at **2**228 (Gateway consumes Hop 1 → max 1 consumer peering hop; a 2nd hop fails preflight229 or returns `SERVFAIL` → `503`, BKI 35). Grant `roles/dns.peer`, `roles/dns.reader`,230 `roles/compute.networkViewer`, and `roles/compute.networkUser` to231 `service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com`232 (and `compute.networkAttachments.use` to caller, BKI 9). Never use authoritative `"."` private root zones (`"."` requires a Cloud DNS Private Forwarding Zone).233- **Private CA `TrustConfig` & Immutable ACT**: Set `egressNetworkConfig.tlsConfig.trustConfig`234 **with `additionalRoots: PUBLICLY_TRUSTED_ROOTS`** (omitting breaks `*.googleapis.com`).235 ACT fields are immutable — create a new ACT and rebind via `PATCH` with236 `updateMask=agentConnectivityTemplate` (BKI 6).237- **Model Armor (`CONTENT_AUTHZ`)**: Ingress (`CLIENT_TO_AGENT`, max 1, ADK238 `streamQuery` only) uses `service-AGENT_PROJECT_NUMBER@gcp-sa-aiplatform-re.iam.gserviceaccount.com`;239 Egress (`AGENT_TO_ANYWHERE`, max 4) uses `service-GATEWAY_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com`.240 Scope `httpRules` `when` to exclude gRPC and streams **>60 seconds** (`Stream removed`, BKI 13).241- **Semantic Governance Policies (SGP)**: Uses `gcp-sa-aiplatform`, non-streaming242 LLM calls only, `.contains()` + `!request.headers['content-type'].startsWith('application/grpc')`243 in `when`, and full-map updates on `PATCH ?updateMask=metadata` so244 `sgpEnforcementMode: "DRY_RUN"` and `sgpUserIntentMatching` are preserved (BKI 38).245- **Native MCP, Identity, Trace, & Teardown**: Set `baseProtocolMethodsOption:246 MATCH_BASE_PROTOCOL_METHODS` on MCP `ALLOW` rules (BKI 39). Orgless trust247 domains use `agents.global.proj-PROJECT_NUMBER.system.id.goog` (`proj-`, never248 `project-`); never grant `roles/aiplatform.expressUser` or project-wide249 `roles/aiplatform.user` to Agent Identities. For Cloud Trace (`TelemetryPolicy`),250 always recommend enabling `cloudtrace.googleapis.com` (`gcloud services enable cloudtrace.googleapis.com`)251 and granting `roles/cloudtrace.agent` to `compute-system`, `gcp-sa-networksecurity`,252 and `gcp-sa-agentgateway` (see `references/agent-gateway.md` §8.3 and `references/field-manual.md` Step 14.4). Tear down in strict 7-step order253 (workloads → UAP bindings → `AuthzPolicy` → extensions → `AgentGateway` → ACT →254 VPC) and use unique gateway IDs if deletion tombstones collide (BKIs 25, 30).255256## Quick Symptom Triage Matrix257258| Symptom / Error | Primary Suspect | Reference |259| :--- | :--- | :--- |260| `403` / `Egress request is not authorized` | Unregistered FQDN, missing `roles/iap.egressor` / UAP binding, or PAB | Rules 4, 6; `references/policies.md` §2; BKIs 4, 7, 8, 15 |261| `HTTP 404` on `HTTP CONNECT` (`HTTPS_PROXY`) | Missing `gcloud beta` L4 `AuthzPolicy` (`networkRules` `snis`) | Rule 13B; `references/agent-gateway.md` §6.3; BKI 34 |262| `HTTP 498` immediately after binding runtime | Enforcing gateway bound before allowlisting essential platform APIs | `references/agent-gateway.md` §2; BKI 37 |263| `500` / `PERMISSION_DENIED` / `Stream removed` | Model Armor SA roles (`gcp-sa-dep` vs `gcp-sa-aiplatform-re`) or 60s stream cap | Rule 14; `references/policies.md` §12; BKI 13 |264| `503` / `NXDOMAIN` / `SERVFAIL` on private host | >2-hop DNS peering, authoritative `"."` zone, Packet Intercept, or SWP | Rules 13A, 14; `references/agent-gateway.md` §3, §6; BKIs 33, 35, 36 |265| `Config validation failed` / `PERMISSION_DENIED` on import | Disabled `agentregistry.googleapis.com`, missing caller/`gcp-sa-agentgateway` IAM, or bad NA URI | Rule 14; BKIs 9, 20, 26 |266| `HTTP 409 ABORTED` / `400 already being used` | Active control-plane LRO mutex (~2 min), tombstone collision, or reverse teardown | `references/agent-gateway.md` §9, §10; BKIs 25, 29, 30 |267| Startup crash (`reasoning_engine_stderr`) | Blocked telemetry/CRM bootstrap, UAP blanket `DENY`, or SGP gRPC interception | Rules 5, 10, 14; BKIs 1, 5, 17, 23, 38 |268269## When to use this skill270271Trigger on Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry,272Agent Identity, IAP/UAP policies, Model Armor, Semantic Governance Policies273(SGP), SWP/Explicit Proxy egress, Private CA `TrustConfig`, Cloud DNS peering,274Cloud Trace `TelemetryPolicy`, or Gemini Enterprise / Agent Runtime / GKE /275Cloud Run gateway bindings. Do **not** use for generic Google Cloud IAM, GCS,276GCE, or networking issues unrelated to the Agent Platform stack.277278## Required context & Hypothesis Rules279280Gather `PROJECT_ID`, `PROJECT_NUMBER`, `LOCATION` (default `us-central1`),281`AGENT_ID`, `AGENT_GATEWAY_NAME`, Agent Identity (`principal://` or282`principalSet://`), exact error + timestamp, and target destination. If missing,283ask or state unknowns using placeholders — never scan across projects or loop284over regions. Before Step 1 queries, formulate **at most 3 hypotheses** anchored285to *"Why did it start failing now?"* (e.g., recent Terraform/config changes).286287## Diagnostic flow288289Follow in order (copy-paste commands in `references/field-manual.md`):290- **Step 0 — Pre-Flight & Access**: Evaluate Rules 1–14. Verify project access291 (`gcloud projects describe $PROJECT_ID`).292- **Step 1 — Agent & Startup Logs**: Classify error (`403`, `404 CONNECT`, `498`,293 `500`, `503`, crash, timeout). Check PSC subnet IPs, ACT `ALL_TRAFFIC` + Cloud294 NAT, and `agentGatewayCard.rootCertificates` CA trust.295- **Step 2 — Gateway Logs**: Query `networkservices.googleapis.com/Gateway`296 (filtering `-httpRequest.requestMethod="CONNECT"` when inspecting L7 requests)297 to find the exact failing SNI/hostname and `serviceExtensionsInfo`.298- **Step 3 — IAP / Model Armor / SGP Logs**: Check `iapPolicyVersion` (`V1` vs299 `V2`), `DRY_RUN` vs enforced, Model Armor callout status, or300 `logs/semantic-governance-policy` verdicts and rationales.301- **Step 4 — Registry State**: Verify exact hostname permutations in302 `gcloud agent-registry {services,endpoints,mcp-servers,agents} list` and303 confirm dual-registry + jurisdiction rules.304- **Step 5 — Identity & Policies**: Check `roles/iap.egressor` via305 `--resource-type=agent-registry` on the child UUID (V1), UAP306 `AccessPolicy`/`PolicyBinding` + `constraints/iam.managed.disableAccessPolicyBinding`307 (V2), PAB overrides, and least-privilege Agent Identity roles (`proj-` for orgless).308- **Step 6 — AuthzPolicy & Extensions**: Verify L4 `networkRules` vs L7309 `httpRules`, MCP `MATCH_BASE_PROTOCOL_METHODS`, Model Armor / SGP `when`310 exclusions, `ExtensionBinding` scope, and policy caps (max 4 egress, max 1 ingress `CONTENT_AUTHZ`).311- **Step 7 — Connectivity, DNS, & TLS**: Verify ACT `TrustConfig`312 (`PUBLICLY_TRUSTED_ROOTS`), `gcp-sa-agentgateway` IAM, <=2 DNS peering hops,313 no Packet Intercept on the consumer VPC, and SWP/Explicit Proxy settings.314- **Step 8 — PrincipalSet & Trace Verification**: Test 1:1 `principal://`315 binding if `principalSet://` lags; for Cloud Trace, enable `cloudtrace.googleapis.com`, verify `v1beta1` `TelemetryPolicy` (`telemetryTarget` referencing `agentGateways/*` and object `parentBasedSampling: {enabled: true, samplingRate: 1.0}`, never string `ENABLED`), and grant `roles/cloudtrace.agent` to `service-<PROJECT_NUMBER>@compute-system.iam.gserviceaccount.com`, `service-<PROJECT_NUMBER>@gcp-sa-networksecurity.iam.gserviceaccount.com`, and `service-<PROJECT_NUMBER>@gcp-sa-agentgateway.iam.gserviceaccount.com`.316317## Tools & Layered References318319Use `mcp__gcloud__run_gcloud_command` (or `gcloud --quiet`),320`mcp__gcloud-observability__list_log_entries` (or `gcloud logging read`),321`mcp__google-dev-knowledge__search_documents` / `get_documents` / `answer_query`,322and `curl` REST GETs, plus bundled references (load only the smallest set needed):323- [references/field-manual.md](references/field-manual.md) — read first for operational commands & log queries324- [references/known-issues.md](references/known-issues.md) — BKIs #1–#13, #15–#39 for fast pattern matches325- [references/agent-gateway.md](references/agent-gateway.md) — topologies, ACT, DNS, TLS, SWP/Explicit Proxy, Trace, teardown326- [references/policies.md](references/policies.md) — L4/L7 `AuthzPolicy`, MCP, UAP v3beta, Model Armor, SGP327- [references/agent-registry.md](references/agent-registry.md) — `services` vs projected `endpoints`, 9 API families, jurisdictions328- [references/agent-identity.md](references/agent-identity.md) — 3-identity delegation, Cloud Run OIDC vs IAP MCP, WIF329330## Output report331332Always produce a read-only diagnostic report using this exact structure:333334```markdown335# Agent Platform Diagnostic — <one-line summary>336337## Context338- Project: <id> (<number>) | Location: <region>339- Agent: <agent_id> | Gateway: <gateway_name>340- Symptom: <exact error message and when it started>341342## Evidence gathered343- Agent log query: <filter, summary>344- Gateway log query: <filter, exact failing hostname / extension status>345- IAP / Policy log query: <filter, decision + enforcement mode>346- Registry & Policy state: <entries, AuthzPolicy bindings, roles/iap.egressor or UAP>347348## Root cause hypothesis349<ranked root cause(s)>350351## Why this fits the evidence352<connect log/config evidence to root cause>353354## Recommended fix355<ordered gcloud / curl / YAML / Terraform changes for the user to apply>356357## What to verify after the fix358<verification queries>359360## Open questions / unknowns361<missing context or unverified items>362363## Appendix: Raw Logs & Verified Links364- **Cloud Logging Filter**: <exact copy-pasteable filter or deep link>365- **Raw Logs**: <untruncated Agent, Gateway, and IAP/Policy log entries>366```367368## Principles369370- **Exact hostname match first**: Compare gateway log hostnames directly against Agent Registry.371- **Layered default-deny**: Registry → Gateway `AuthzPolicy` → Extension → IAP/UAP/Model Armor/SGP → PAB (PAB overrides IAM Allow).372- **L4 bypass vs L7 governance**: L4 `networkRules` enables `HTTP CONNECT` explicit proxies (`:443`) but bypasses L7 IAP, Model Armor, and SGP for matched SNIs.373- **Read-only, single-region, non-interactive**: Pull evidence first, never mutate resources, check `us-central1` unless logs indicate otherwise, and pass `--quiet`.374375## Supporting Links376377- [Agent Gateway Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/agent-gateway-overview.md.txt)378- [Configure VPC Connectivity & DNS](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/set-up-vpc-connectivity.md.txt)379- [Delegate Authorization & MCP](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/delegate-authorization.md.txt)380- [Configure Model Armor](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/configure-model-armor.md.txt)381- [Configure Semantic Governance](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/configure-semantic-governance.md.txt)382- [Monitor & Trace Agent Gateway](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/monitor-agent-gateway.md.txt)383- [Deploy Agent Gateway Runtime](https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/agent-gateway-runtime-deploy.md.txt)384- [Troubleshoot Agent Gateway](https://docs.cloud.google.com/gemini-enterprise-agent-platform/troubleshooting/troubleshoot-agent-gateway.md.txt)385- [Troubleshoot IAM Policies (UAP)](https://docs.cloud.google.com/gemini-enterprise-agent-platform/troubleshooting/troubleshoot-iam-policies-uap.md.txt)