Troubleshoot GKE TPU performance degradation with ML Diagnostics Workload Monitoring
Diagnose and mitigate Cloud TPU training throughput drops and step-time
regressions (15%+ drop in TPU duty cycle) on Google Kubernetes Engine (GKE) by
correlating ML Diagnostics Workload Monitoring MonitoredEvent analyzer
reports with 1-minute Cloud Monitoring system metrics and GKE node topology
labels.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-performance-degradation)"
and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-performance-degradation) to User-Agent headers.
Prerequisites
- Tools: Install the
Google Cloud SDK (
gcloud with
alpha component for gcloud alpha mldiagnostics) and kubectl.
- Cloud Billing & Project Configuration: Verify an active billing account is
linked (
gcloud billing projects describe {project_id}), authenticate
(gcloud auth login), set the target project (gcloud config set project {project_id}), and ensure container.googleapis.com,
monitoring.googleapis.com, and hypercomputecluster.googleapis.com are
enabled.
- Supported workloads and GKE versions: Google Cloud ML Diagnostics only
supports JAX on TPUs (see
ML Diagnostics platform).
Workload Monitoring is enabled by default, supports the
jobset and job GKE
job types, and is compatible with GKE versions 1.36.0-gke.4681000 and later,
as stated in
Configure GKE for ML Diagnostics.
If a workload uses another framework (such as PyTorch) or another custom
resource type, gcloud alpha mldiagnostics won't list ML runs or monitored
events for it. On-demand profiling (Step 4 Path A) additionally requires the
cluster setup described in that document.
- Required IAM Roles:
- Cluster Director Editor (
roles/hypercomputecluster.editor), the role
listed in the "IAM permissions" section of
ML Diagnostics platform,
for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored
events, and on-demand profiler sessions)
- Monitoring Viewer (
roles/monitoring.viewer) for the PromQL queries in Step
2
- Kubernetes Engine Viewer (
roles/container.viewer) for the kubectl get nodes query in Step 3
- For remediation (
[High Risk] steps): Kubernetes Engine Cluster Admin
(roles/container.clusterAdmin)
- Reference Documentation:
- ML Diagnostics platform
(Sections: "IAM permissions")
- Configure GKE for ML Diagnostics
(Sections: "Set up with gcloud CLI, Google Cloud console, or Terraform",
"Manual installation", "Connection-operator")
- Workload monitoring with ML Diagnostics
(Sections: "Workload Monitoring and analyzers", "HBM Capacity Analyzer",
"Host Memory Utilization Analyzer", "CPU Utilization Analyzer", "Access
Workload Monitoring information through the API", "List all monitored
events", "System Metrics")
- Get started with the ML Diagnostics CLI
(Sections: "List machine learning runs", "Monitored-events commands", "List
profiler targets", "Capture on-demand profiler sessions")
- Scale container resource requests and limits
(Sections: "Identify workloads without resource requests or limits")
- Auto-repair nodes
(Sections: "Repair criteria", "Verify node auto-repair is enabled for a
Standard node pool", "Enable auto-repair for an existing Standard node
pool", "Node auto repair in TPU slice nodes")
- Deploy TPU workloads in GKE Standard
(Sections: "Configure auto repair for TPU slice nodes")
- Get support
(Sections: "Before you contact support")
Read-only rule: Run read-only diagnostic commands only. Never drain,
delete, or re-create nodes, or run any other command that changes the cluster.
Give the user any fix to apply themselves.
When you recommend a fix, link the doc section that describes it.
Analyzer routing overview
By default, Workload Monitoring treats a 15% drop in TPU duty cycle as a
performance degradation, raises a MonitoredEvent (type: PERFORMANCE_DEGRADATION), and runs its analyzers. Consult the
Workload Monitoring and analyzers
section in the official Cloud TPU documentation for the complete analyzer
definitions, detection criteria, and subsections, and route remediation by
category:
- Category A: workload resource bottlenecks (never cordon or replace nodes):
- When the firing analyzer (
detectionState: "DETECTED") indicates a workload
memory or host CPU capacity bottleneck (see the "HBM Capacity Analyzer",
"Host Memory Utilization Analyzer", and "CPU Utilization Analyzer" sections
of
Workload monitoring with ML Diagnostics),
route to
Path A: Workload resource bottlenecks
(workload optimization and profiling).
- Category B: infrastructure and network fabric throttling (culprit nodes):
- When the firing analyzer (
detectionState: "DETECTED") indicates an
interconnect, thermal/power throttling, memory bandwidth, or Top-of-Rack
(ToR) network fault on specific instances (see
Workload Monitoring and analyzers),
route to
Path B: Infrastructure or network fabric throttling
(map the culprit Compute Engine instance IDs to GKE nodes, handle node
repair through GKE, and escalate to Google Cloud Support).
- Category C:
PERFORMANCE_DEGRADATION event fired, but no analyzer reports
detectionState: "DETECTED":
- When a
PERFORMANCE_DEGRADATION event exists (as in the sample output in
List all monitored events,
where analyzerReports entries have detectionState: "NOT_DETECTED"),
route to
Path C: Event fired with no DETECTED analyzer.
Diagnostic workflow
Step 0: Collect context and set the investigation window [Low Risk]
Collect the target parameters. By default, query a 60-minute window `[T - 30m, T
{project_id}: Google Cloud project ID
{location}: Google Cloud region where the ML run and GKE cluster reside (for
example, us-central1)
{cluster_name}: GKE cluster name
{workload_name} / {ml_run_id}: JobSet / workload name or ML Diagnostics
run ID
{issue_time}: Timestamp when throughput degradation was observed (T,
ISO-8601 UTC)
{start_time}: T - 30m
{end_time}: T + 30m
Step 1: Query ML runs and monitoredEvents [Low Risk]
- List active or recent ML runs: Give the user
gcloud alpha mldiagnostics machine-learning-run list with a link to
List machine learning runs
in the ML Diagnostics CLI reference, and locate {ml_run_id} matching
{workload_name}.
- List performance degradation events: Give the user
gcloud alpha mldiagnostics monitored-events list with a link to
Monitored-events commands,
or the hypercomputecluster.googleapis.com/v1alpha API with a link to
Access Workload Monitoring information through the API,
to check for PERFORMANCE_DEGRADATION events during [{start_time}, {end_time}].
- Describe the
MonitoredEvent: Give the user gcloud alpha mldiagnostics monitored-events describe with a link to the same
Monitored-events commands
section, and inspect the analyzerReports array (analyzer,
detectionState, details, and recommendedActions).
- If a
PERFORMANCE_DEGRADATION event fired, run Step 2 to corroborate with
the 1-minute system metrics; if none of its analyzerReports entries has
detectionState: "DETECTED", follow
Path C: Event fired with no DETECTED analyzer.
- If no
PERFORMANCE_DEGRADATION event exists and duty cycle is steady in
Step 2, rule out TPU performance degradation by following
Path D: Healthy telemetry.
Step 2: Correlate with 1-minute Cloud Monitoring system metrics [Low Risk]
Consult the
System Metrics
section in the Workload Monitoring documentation for the 1-minute Cloud
Monitoring metrics exported for TPU and host devices, and run read-only PromQL
queries over [{start_time}, {end_time}] to corroborate the analyzer report:
# 1. Node TPU duty cycle (look for the drop on the affected nodes)
kubernetes_io:node_accelerator_duty_cycle{
monitored_resource="k8s_node",
project_id="{project_id}",
cluster_name="{cluster_name}"
}
# 2. HBM utilization ratio by node (around 0.90 is approaching the limit)
sum by (node_name) (
kubernetes_io:node_accelerator_memory_used{
monitored_resource="k8s_node",
project_id="{project_id}",
cluster_name="{cluster_name}"
}
)
/
sum by (node_name) (
kubernetes_io:node_accelerator_memory_total{
monitored_resource="k8s_node",
project_id="{project_id}",
cluster_name="{cluster_name}"
}
)
# 3. Host memory and CPU allocatable utilization
kubernetes_io:node_memory_allocatable_utilization{
monitored_resource="k8s_node",
project_id="{project_id}",
cluster_name="{cluster_name}"
}
kubernetes_io:node_cpu_allocatable_utilization{
monitored_resource="k8s_node",
project_id="{project_id}",
cluster_name="{cluster_name}"
}
Step 3: Map culprit instance IDs to GKE nodes and topology [Low Risk]
When an infrastructure analyzer reports culprit numeric Compute Engine instance
IDs in details or recommendedActions, map those numeric instance IDs to GKE
Node names and physical topology blocks using this read-only kubectl query
inspecting container.googleapis.com/instance_id:
kubectl get nodes -l cloud.google.com/gke-tpu-accelerator \
-o jsonpath='{range .items[*]}{.metadata.name}{"\tinstance_id="}{.metadata.annotations.container\.googleapis\.com/instance_id}{"\tblock="}{.metadata.labels.cloud\.google\.com/gce-topology-block}{"\tsubblock="}{.metadata.labels.cloud\.google\.com/gce-topology-subblock}{"\thost="}{.metadata.labels.cloud\.google\.com/gce-topology-host}{"\n"}{end}'
Step 4: Remediation by analyzer category
Load and follow only the reference file that matches the analyzer category from
Step 1:
- Path A (workload resource bottlenecks — HBM, host memory, or host CPU
utilization): Read
Path A: Workload resource bottlenecks.
- Path B (infrastructure, thermal, ICI, memory bandwidth, or network fabric
throttling): Read
Path B: Infrastructure or network fabric throttling.
- Path C (
PERFORMANCE_DEGRADATION event fired with all analyzers
NOT_DETECTED): Read
Path C: Event fired with no DETECTED analyzer.
- Path D (no
PERFORMANCE_DEGRADATION events and steady telemetry): Read
Path D: Healthy telemetry.
Guardrails
- Never change GKE-managed instance groups or VMs through Compute Engine:
Don't run
gcloud compute instance-groups managed commands, such as
delete, on a node pool's managed instance group. Handle nodes through GKE,
as described in
Path B: Infrastructure or network fabric throttling.
- Never cordon nodes for workload resource saturation: If only the HBM
capacity, host memory, or host CPU utilization analyzers detected an issue,
don't cordon or replace nodes.
1---2description: Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without crashing pods, when `PERFORMANCE_DEGRADATION` monitored events fire, or when triaging slow multi-slice training steps. Don't use for complete multi-slice XLA execution stalls with `HANG_DETECTED` logs (use gke-ai-troubleshooting-tpu-mxla-hang) or pod eviction/interruption restarts (use gke-ai-troubleshooting-jobset-interruption).3metadata:4 source: https://skillmd.com/skills/h3y6e/gke-ai-troubleshooting-tpu-performance-degradation5 category: AiAndMachineLearning6 github-path: skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation7 github-ref: refs/heads/main8 github-repo: https://github.com/google/skills9 github-tree-sha: e8172480210fbd64147b66ca93d0f0717ac7dabf10 version: 1.0.111name: gke-ai-troubleshooting-tpu-performance-degradation12---1314# Troubleshoot GKE TPU performance degradation with ML Diagnostics Workload Monitoring1516Diagnose and mitigate Cloud TPU training throughput drops and step-time17regressions (`15%+` drop in TPU duty cycle) on Google Kubernetes Engine (GKE) by18correlating **ML Diagnostics Workload Monitoring** `MonitoredEvent` analyzer19reports with **1-minute Cloud Monitoring system metrics** and GKE node topology20labels.2122---2324## Attribution2526**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you27MUST read [Usage Attribution](references/attribution.md). For commands executed28directly in the terminal (never in generated scripts, Terraform, or user code),29prefix `gcloud` with30`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-performance-degradation)"`31and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-performance-degradation)` to `User-Agent` headers.3233## Prerequisites3435- **Tools**: Install the36 [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) (`gcloud` with37 `alpha` component for `gcloud alpha mldiagnostics`) and `kubectl`.38- **Cloud Billing & Project Configuration**: Verify an active billing account is39 linked (`gcloud billing projects describe {project_id}`), authenticate40 (`gcloud auth login`), set the target project (`gcloud config set project41 {project_id}`), and ensure `container.googleapis.com`,42 `monitoring.googleapis.com`, and `hypercomputecluster.googleapis.com` are43 enabled.44- **Supported workloads and GKE versions**: Google Cloud ML Diagnostics only45 supports JAX on TPUs (see46 [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)).47 Workload Monitoring is enabled by default, supports the `jobset` and `job` GKE48 job types, and is compatible with GKE versions `1.36.0-gke.4681000` and later,49 as stated in50 [Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt).51 If a workload uses another framework (such as PyTorch) or another custom52 resource type, `gcloud alpha mldiagnostics` won't list ML runs or monitored53 events for it. On-demand profiling (Step 4 Path A) additionally requires the54 cluster setup described in that document.55- **Required IAM Roles**:56 - Cluster Director Editor (`roles/hypercomputecluster.editor`), the role57 listed in the "IAM permissions" section of58 [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt),59 for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored60 events, and on-demand profiler sessions)61 - Monitoring Viewer (`roles/monitoring.viewer`) for the PromQL queries in Step62 263 - Kubernetes Engine Viewer (`roles/container.viewer`) for the `kubectl get64 nodes` query in Step 365 - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin66 (`roles/container.clusterAdmin`)67- **Reference Documentation**:68 - [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)69 (Sections: "IAM permissions")70 - [Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt)71 (Sections: "Set up with gcloud CLI, Google Cloud console, or Terraform",72 "Manual installation", "Connection-operator")73 - [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)74 (Sections: "Workload Monitoring and analyzers", "HBM Capacity Analyzer",75 "Host Memory Utilization Analyzer", "CPU Utilization Analyzer", "Access76 Workload Monitoring information through the API", "List all monitored77 events", "System Metrics")78 - [Get started with the ML Diagnostics CLI](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)79 (Sections: "List machine learning runs", "Monitored-events commands", "List80 profiler targets", "Capture on-demand profiler sessions")81 - [Scale container resource requests and limits](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/vertical-pod-autoscaling.md.txt)82 (Sections: "Identify workloads without resource requests or limits")83 - [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)84 (Sections: "Repair criteria", "Verify node auto-repair is enabled for a85 Standard node pool", "Enable auto-repair for an existing Standard node86 pool", "Node auto repair in TPU slice nodes")87 - [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)88 (Sections: "Configure auto repair for TPU slice nodes")89 - [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)90 (Sections: "Before you contact support")9192> **Read-only rule**: Run read-only diagnostic commands only. Never drain,93> delete, or re-create nodes, or run any other command that changes the cluster.94> Give the user any fix to apply themselves.9596When you recommend a fix, link the doc section that describes it.9798---99100## Analyzer routing overview101102By default, Workload Monitoring treats a 15% drop in TPU duty cycle as a103performance degradation, raises a `MonitoredEvent` (`type:104PERFORMANCE_DEGRADATION`), and runs its analyzers. Consult the105[Workload Monitoring and analyzers](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)106section in the official Cloud TPU documentation for the complete analyzer107definitions, detection criteria, and subsections, and route remediation by108category:109110- **Category A: workload resource bottlenecks (never cordon or replace nodes)**:111 - When the firing analyzer (`detectionState: "DETECTED"`) indicates a workload112 memory or host CPU capacity bottleneck (see the "HBM Capacity Analyzer",113 "Host Memory Utilization Analyzer", and "CPU Utilization Analyzer" sections114 of115 [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)),116 route to117 [Path A: Workload resource bottlenecks](references/path-a-workload-bottlenecks.md)118 (workload optimization and profiling).119- **Category B: infrastructure and network fabric throttling (culprit nodes)**:120 - When the firing analyzer (`detectionState: "DETECTED"`) indicates an121 interconnect, thermal/power throttling, memory bandwidth, or Top-of-Rack122 (ToR) network fault on specific instances (see123 [Workload Monitoring and analyzers](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)),124 route to125 [Path B: Infrastructure or network fabric throttling](references/path-b-infrastructure-throttling.md)126 (map the culprit Compute Engine instance IDs to GKE nodes, handle node127 repair through GKE, and escalate to Google Cloud Support).128- **Category C: `PERFORMANCE_DEGRADATION` event fired, but no analyzer reports129 `detectionState: "DETECTED"`**:130 - When a `PERFORMANCE_DEGRADATION` event exists (as in the sample output in131 [List all monitored events](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt),132 where `analyzerReports` entries have `detectionState: "NOT_DETECTED"`),133 route to134 [Path C: Event fired with no DETECTED analyzer](references/path-c-no-detected-analyzer.md).135136---137138## Diagnostic workflow139140### Step 0: Collect context and set the investigation window `[Low Risk]`141142Collect the target parameters. By default, query a 60-minute window `[T - 30m, T143+ 30m]` around `{issue_time}`:144145- `{project_id}`: Google Cloud project ID146- `{location}`: Google Cloud region where the ML run and GKE cluster reside (for147 example, `us-central1`)148- `{cluster_name}`: GKE cluster name149- `{workload_name}` / `{ml_run_id}`: JobSet / workload name or ML Diagnostics150 run ID151- `{issue_time}`: Timestamp when throughput degradation was observed (`T`,152 ISO-8601 UTC)153- `{start_time}`: `T - 30m`154- `{end_time}`: `T + 30m`155156---157158### Step 1: Query ML runs and `monitoredEvents` `[Low Risk]`1591601. **List active or recent ML runs**: Give the user `gcloud alpha mldiagnostics161 machine-learning-run list` with a link to162 [List machine learning runs](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)163 in the ML Diagnostics CLI reference, and locate `{ml_run_id}` matching164 `{workload_name}`.1652. **List performance degradation events**: Give the user `gcloud alpha166 mldiagnostics monitored-events list` with a link to167 [Monitored-events commands](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt),168 or the `hypercomputecluster.googleapis.com/v1alpha` API with a link to169 [Access Workload Monitoring information through the API](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt),170 to check for `PERFORMANCE_DEGRADATION` events during `[{start_time},171 {end_time}]`.1723. **Describe the `MonitoredEvent`**: Give the user `gcloud alpha mldiagnostics173 monitored-events describe` with a link to the same174 [Monitored-events commands](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)175 section, and inspect the `analyzerReports` array (`analyzer`,176 `detectionState`, `details`, and `recommendedActions`).177 - If a `PERFORMANCE_DEGRADATION` event fired, run Step 2 to corroborate with178 the 1-minute system metrics; if none of its `analyzerReports` entries has179 `detectionState: "DETECTED"`, follow180 [Path C: Event fired with no DETECTED analyzer](references/path-c-no-detected-analyzer.md).181 - If no `PERFORMANCE_DEGRADATION` event exists and duty cycle is steady in182 Step 2, **rule out** TPU performance degradation by following183 [Path D: Healthy telemetry](references/path-d-healthy-telemetry.md).184185---186187### Step 2: Correlate with 1-minute Cloud Monitoring system metrics `[Low Risk]`188189Consult the190[System Metrics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)191section in the Workload Monitoring documentation for the 1-minute Cloud192Monitoring metrics exported for TPU and host devices, and run read-only PromQL193queries over `[{start_time}, {end_time}]` to corroborate the analyzer report:194195```promql196# 1. Node TPU duty cycle (look for the drop on the affected nodes)197kubernetes_io:node_accelerator_duty_cycle{198 monitored_resource="k8s_node",199 project_id="{project_id}",200 cluster_name="{cluster_name}"201}202203# 2. HBM utilization ratio by node (around 0.90 is approaching the limit)204sum by (node_name) (205 kubernetes_io:node_accelerator_memory_used{206 monitored_resource="k8s_node",207 project_id="{project_id}",208 cluster_name="{cluster_name}"209 }210)211/212sum by (node_name) (213 kubernetes_io:node_accelerator_memory_total{214 monitored_resource="k8s_node",215 project_id="{project_id}",216 cluster_name="{cluster_name}"217 }218)219220# 3. Host memory and CPU allocatable utilization221kubernetes_io:node_memory_allocatable_utilization{222 monitored_resource="k8s_node",223 project_id="{project_id}",224 cluster_name="{cluster_name}"225}226227kubernetes_io:node_cpu_allocatable_utilization{228 monitored_resource="k8s_node",229 project_id="{project_id}",230 cluster_name="{cluster_name}"231}232```233234---235236### Step 3: Map culprit instance IDs to GKE nodes and topology `[Low Risk]`237238When an infrastructure analyzer reports culprit numeric Compute Engine instance239IDs in `details` or `recommendedActions`, map those numeric instance IDs to GKE240`Node` names and physical topology blocks using this read-only `kubectl` query241inspecting `container.googleapis.com/instance_id`:242243```bash244kubectl get nodes -l cloud.google.com/gke-tpu-accelerator \245 -o jsonpath='{range .items[*]}{.metadata.name}{"\tinstance_id="}{.metadata.annotations.container\.googleapis\.com/instance_id}{"\tblock="}{.metadata.labels.cloud\.google\.com/gce-topology-block}{"\tsubblock="}{.metadata.labels.cloud\.google\.com/gce-topology-subblock}{"\thost="}{.metadata.labels.cloud\.google\.com/gce-topology-host}{"\n"}{end}'246```247248---249250### Step 4: Remediation by analyzer category251252Load and follow only the reference file that matches the analyzer category from253Step 1:254255- **Path A (workload resource bottlenecks — HBM, host memory, or host CPU256 utilization)**: Read257 [Path A: Workload resource bottlenecks](references/path-a-workload-bottlenecks.md).258- **Path B (infrastructure, thermal, ICI, memory bandwidth, or network fabric259 throttling)**: Read260 [Path B: Infrastructure or network fabric throttling](references/path-b-infrastructure-throttling.md).261- **Path C (`PERFORMANCE_DEGRADATION` event fired with all analyzers262 `NOT_DETECTED`)**: Read263 [Path C: Event fired with no DETECTED analyzer](references/path-c-no-detected-analyzer.md).264- **Path D (no `PERFORMANCE_DEGRADATION` events and steady telemetry)**: Read265 [Path D: Healthy telemetry](references/path-d-healthy-telemetry.md).266267---268269## Guardrails2702711. **Never change GKE-managed instance groups or VMs through Compute Engine**:272 Don't run `gcloud compute instance-groups managed` commands, such as273 `delete`, on a node pool's managed instance group. Handle nodes through GKE,274 as described in275 [Path B: Infrastructure or network fabric throttling](references/path-b-infrastructure-throttling.md).2762. **Never cordon nodes for workload resource saturation**: If only the HBM277 capacity, host memory, or host CPU utilization analyzers detected an issue,278 don't cordon or replace nodes.