Troubleshoot GKE TPU multi-slice hangs with the MXLA Hang Analyzer
Diagnose Cloud TPU multi-slice training hangs on Google Kubernetes Engine (GKE)
by correlating Megascale HANG_DETECTED logs and ML Diagnostics Workload
Monitoring Megascale XLA (MXLA) Hang Analyzer reports with 1-minute
multi-slice latency metrics (kubernetes.io/container/multislice/*) and GKE
node topology labels.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)"
and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang) to User-Agent headers.
Prerequisites
- Tools: Install the
Google Cloud SDK (
gcloud with
alpha component for gcloud alpha mldiagnostics) and kubectl.
- Cloud Billing & Project Configuration: Verify an active billing account is
linked (
gcloud billing projects describe {project_id}), authenticate
(gcloud auth login), set the target project (gcloud config set project {project_id}), and ensure container.googleapis.com,
logging.googleapis.com, monitoring.googleapis.com, and
hypercomputecluster.googleapis.com are enabled.
- Supported workloads and versions: Google Cloud ML Diagnostics only
supports JAX on TPUs (see
ML Diagnostics platform).
Workload Monitoring is enabled by default, supports the
jobset and job GKE
job types, and is compatible with GKE versions 1.36.0-gke.4681000 and later
(Configure GKE for ML Diagnostics,
which also covers the cluster setup needed for on-demand profiling in Step 4
Path B). If a workload uses another framework (such as PyTorch) or another
custom resource type, gcloud alpha mldiagnostics won't list ML runs or
monitored events for it. The Megascale XLA hang analyzer and Megascale XLA
metrics require LibTPU 0.40.0 or later (see "Get started" in
Workload monitoring with ML Diagnostics).
- Required IAM Roles:
- Cluster Director Editor (
roles/hypercomputecluster.editor), the role
listed in the "IAM permissions" section of
ML Diagnostics platform,
for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored
events, and on-demand profiler sessions)
- Monitoring Viewer (
roles/monitoring.viewer) for the PromQL queries in Step
2
- Logs Viewer (
roles/logging.viewer) for the Cloud Logging query in Step 1
- Kubernetes Engine Viewer (
roles/container.viewer) for the kubectl get nodes query in Step 3
- For remediation (
[High Risk] steps): Kubernetes Engine Cluster Admin
(roles/container.clusterAdmin)
- Reference Documentation:
- ML Diagnostics platform
(Sections: "IAM permissions")
- Configure GKE for ML Diagnostics
(Sections: "Set up with gcloud CLI, Google Cloud console, or Terraform",
"Manual installation", "Connection-operator")
- Workload monitoring with ML Diagnostics
(Sections: "Get started", "Megascale XLA (MXLA) Hang Analyzer", "Access
Workload Monitoring information through the API", "System Metrics")
- Get started with the ML Diagnostics CLI
(Sections: "List machine learning runs", "Monitored-events commands", "List
profiler targets", "Capture on-demand profiler sessions")
- Get support
(Sections: "Before you contact support")
- Auto-repair nodes
(Sections: "Repair criteria", "Verify node auto-repair is enabled for a
Standard node pool", "Enable auto-repair for an existing Standard node
pool", "Node auto repair in TPU slice nodes")
- Deploy TPU workloads in GKE Standard
(Sections: "Configure auto repair for TPU slice nodes")
- Dump HLO Computations (OpenXLA)
Read-only rule: Run read-only diagnostic commands only. Never drain,
delete, or re-create nodes, or run any other command that changes the cluster.
Give the user any fix to apply themselves.
When you recommend a fix, link the doc section that describes it.
MXLA Hang Analyzer routing overview
A Megascale hang occurs when a multi-slice worker has waited on a Megascale
communication operation for a set timeout period. The TPU logs then show a
Megascale HANG_DETECTED message. HANG_DETECTED is a catch-all signal that
the workload isn't progressing, and the cause can be in software or in hardware.
When a hang occurs, ML Diagnostics runs the Megascale XLA (MXLA) Hang Analyzer, which reports the likely cause as a code. Consult the
Megascale XLA (MXLA) Hang Analyzer
section for the definition and recommended action of each code, and route by
category:
- Category A: compiler or HLO divergence (never cordon or replace nodes):
- When the analyzer reports different HLO modules, inconsistent HLO
compilation, or an inconsistent launch order across VMs (such as
FINGERPRINT_MISMATCH), route to
Path A: Compiler or HLO divergence.
- Category B: host program queueing or data input stall (never cordon or
replace nodes):
- When the analyzer reports that VMs aren't queuing programs to the TPU or
that data input stalled (such as
DATA_INPUT_STALL), route to
Path B: Host program queueing or data input stall.
- Category C: hardware or network faults on specific instances:
- When the analyzer attributes the hang to a TPU chip, SparseCore, ICI, or DCN
networking issue, or to an unrecoverable error on specific instances, route
to
Path C: Hardware or network faults.
- Category D: hang signal or event exists, but the analyzer is
NOT_DETECTED
or reports UNKNOWN:
- When
HANG_DETECTED logs or a hang monitored event fired, but the analyzer
report has detectionState: "NOT_DETECTED" or reports UNKNOWN ("The MXLA
hang was detected but a potential cause is not determined"), route to
Path D: No DETECTED analyzer or UNKNOWN cause.
Diagnostic workflow
Step 0: Collect context and set the investigation window [Low Risk]
Collect the target parameters. By default, query a 60-minute window `[T - 30m, T
{project_id}: Google Cloud project ID
{location}: Google Cloud region where the ML run and GKE cluster reside (for
example, us-central1)
{cluster_name}: GKE cluster name
{namespace} / {workload_name}: Kubernetes namespace and JobSet/Pod prefix
{ml_run_id}: ML Diagnostics run ID
{issue_time}: Timestamp when the hang occurred (T, ISO-8601 UTC)
{start_time}: T - 30m
{end_time}: T + 30m
Step 1: Query HANG_DETECTED logs and hang events [Low Risk]
- Check GKE container logs for
HANG_DETECTED (read-only Cloud Logging
LQL): Query k8s_container logs over [{start_time}, {end_time}] to
confirm HANG_DETECTED and identify the first stalled pods:
resource.type="k8s_container"
resource.labels.project_id="{project_id}"
resource.labels.cluster_name="{cluster_name}"
"HANG_DETECTED"
timestamp >= "{start_time}" AND timestamp <= "{end_time}"
- List active or recent ML runs: Follow the
List machine learning runs
section of the ML Diagnostics CLI reference (
gcloud alpha mldiagnostics machine-learning-run list) to identify {ml_run_id}.
- List and describe hang monitored events: Follow the
Monitored-events commands
section (
gcloud alpha mldiagnostics monitored-events list and gcloud alpha mldiagnostics monitored-events describe) or
Access Workload Monitoring information through the API
to inspect the Megascale XLA (MXLA) Hang Analyzer report (detectionState,
details, and recommendedActions).
- If
HANG_DETECTED logs or a hang monitored event fired, run Step 2 to
correlate with the 1-minute metrics; if no analyzer reports
detectionState: "DETECTED" or the analyzer reports UNKNOWN, follow
Path D: No DETECTED analyzer or UNKNOWN cause.
- If no
HANG_DETECTED log or hang monitored event exists and multi-slice
latencies are normal in Step 2, rule out an MXLA hang by following
Path E: Healthy telemetry.
Step 2: Correlate with 1-minute multi-slice and TPU metrics [Low Risk]
Consult the
System Metrics
section of the Workload Monitoring guide for the 1-minute multi-slice network
(kubernetes.io/container/multislice/network/*), multi-slice accelerator
(kubernetes.io/container/multislice/accelerator/*), and node duty-cycle
(kubernetes.io/node/accelerator/duty_cycle) metrics, and run read-only PromQL
queries over [{start_time}, {end_time}]:
# 1. P95 multi-slice collective end-to-end latency by pod
histogram_quantile(
0.95,
sum by (pod_name, le) (
rate(kubernetes_io:container_multislice_network_collective_end_to_end_latencies_bucket{
monitored_resource="k8s_container",
project_id="{project_id}",
cluster_name="{cluster_name}"
}[5m])
)
)
# 2. P95 multi-slice DCN transfer latency by pod
histogram_quantile(
0.95,
sum by (pod_name, le) (
rate(kubernetes_io:container_multislice_network_dcn_transfer_latencies_bucket{
monitored_resource="k8s_container",
project_id="{project_id}",
cluster_name="{cluster_name}"
}[5m])
)
)
# 3. P95 host-to-device transfer latency by pod
histogram_quantile(
0.95,
sum by (pod_name, le) (
rate(kubernetes_io:container_multislice_accelerator_host_to_device_transfer_latencies_bucket{
monitored_resource="k8s_container",
project_id="{project_id}",
cluster_name="{cluster_name}"
}[5m])
)
)
# 4. Node TPU duty cycle (Workload Monitoring detects a hang as a prolonged
# period of minimal to no TPU activity)
kubernetes_io:node_accelerator_duty_cycle{
monitored_resource="k8s_node",
project_id="{project_id}",
cluster_name="{cluster_name}"
}
Step 3: Map culprit Compute Engine instance IDs to GKE nodes [Low Risk]
When the Megascale XLA (MXLA) Hang Analyzer reports culprit numeric Compute
Engine instance IDs in details or recommendedActions, map those numeric IDs
to GKE Node names and physical topology blocks using this read-only kubectl
query inspecting container.googleapis.com/instance_id:
kubectl get nodes -l cloud.google.com/gke-tpu-accelerator \
-o jsonpath='{range .items[*]}{.metadata.name}{"\tinstance_id="}{.metadata.annotations.container\.googleapis\.com/instance_id}{"\tblock="}{.metadata.labels.cloud\.google\.com/gce-topology-block}{"\tsubblock="}{.metadata.labels.cloud\.google\.com/gce-topology-subblock}{"\thost="}{.metadata.labels.cloud\.google\.com/gce-topology-host}{"\n"}{end}'
Step 4: Remediation by root-cause category
Load and follow only the reference file that matches the root-cause category
from Step 1:
- Path A (compiler or HLO divergence — different HLO modules, inconsistent HLO
compilation, or inconsistent launch order across VMs, such as
FINGERPRINT_MISMATCH): Read
Path A: Compiler or HLO divergence.
- Path B (host program queueing or data input stall —
PROGRAM_NOT_QUEUED or
DATA_INPUT_STALL): Read
Path B: Host program queueing or data input stall.
- Path C (hardware, SparseCore, ICI, DCN networking faults, or
UNRECOVERABLE_ERROR on specific instances): Read
Path C: Hardware or network faults.
- Path D (
HANG_DETECTED logs or hang event fired, but the analyzer is
NOT_DETECTED or reports UNKNOWN): Read
Path D: No DETECTED analyzer or UNKNOWN cause.
- Path E (no
HANG_DETECTED logs or hang events, and steady telemetry):
Read Path E: Healthy telemetry.
Guardrails
- Never change GKE-managed instance groups or VMs through Compute Engine:
Don't run
gcloud compute instance-groups managed commands, such as
delete, on a node pool's managed instance group. Handle nodes through GKE,
as described in
Path C: Hardware or network faults.
- Never cordon or replace nodes for compiler divergence or input stalls: If
the analyzer reports a compiler or HLO divergence, a program queueing issue,
or a data input stall, don't cordon or replace TPU nodes. Follow
Path A: Compiler or HLO divergence
or
Path B: Host program queueing or data input stall
instead.
1---2name: gke-ai-troubleshooting-tpu-mxla-hang3description: >-4 Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED`5 logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA)6 Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute7 Cloud Monitoring multi-slice latency metrics8 (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO9 launch divergence and host data-input stalls from TPU chip, SparseCore, ICI,10 or network fabric faults. Use when multi-slice TPU training jobs freeze11 without progressing steps, emit `HANG_DETECTED`, or stall in collective12 operations. Don't use for gradual step-time throughput drops without hangs13 (use gke-ai-troubleshooting-tpu-performance-degradation) or pod14 preemption/eviction restarts (use gke-ai-troubleshooting-jobset-interruption).15metadata:16 source: https://skillmd.com/skills/google/gke-ai-troubleshooting-tpu-mxla-hang17 version: "1.0.1"18 category: AiAndMachineLearning19---2021# Troubleshoot GKE TPU multi-slice hangs with the MXLA Hang Analyzer2223Diagnose Cloud TPU multi-slice training hangs on Google Kubernetes Engine (GKE)24by correlating Megascale `HANG_DETECTED` logs and **ML Diagnostics Workload25Monitoring** `Megascale XLA (MXLA) Hang Analyzer` reports with **1-minute26multi-slice latency metrics** (`kubernetes.io/container/multislice/*`) and GKE27node topology labels.2829---3031## Attribution3233**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you34MUST read [Usage Attribution](references/attribution.md). For commands executed35directly in the terminal (never in generated scripts, Terraform, or user code),36prefix `gcloud` with37`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)"`38and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)` to `User-Agent` headers.3940## Prerequisites4142- **Tools**: Install the43 [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) (`gcloud` with44 `alpha` component for `gcloud alpha mldiagnostics`) and `kubectl`.45- **Cloud Billing & Project Configuration**: Verify an active billing account is46 linked (`gcloud billing projects describe {project_id}`), authenticate47 (`gcloud auth login`), set the target project (`gcloud config set project48 {project_id}`), and ensure `container.googleapis.com`,49 `logging.googleapis.com`, `monitoring.googleapis.com`, and50 `hypercomputecluster.googleapis.com` are enabled.51- **Supported workloads and versions**: Google Cloud ML Diagnostics only52 supports JAX on TPUs (see53 [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)).54 Workload Monitoring is enabled by default, supports the `jobset` and `job` GKE55 job types, and is compatible with GKE versions `1.36.0-gke.4681000` and later56 ([Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt),57 which also covers the cluster setup needed for on-demand profiling in Step 458 Path B). If a workload uses another framework (such as PyTorch) or another59 custom resource type, `gcloud alpha mldiagnostics` won't list ML runs or60 monitored events for it. The Megascale XLA hang analyzer and Megascale XLA61 metrics require LibTPU `0.40.0` or later (see "Get started" in62 [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)).63- **Required IAM Roles**:64 - Cluster Director Editor (`roles/hypercomputecluster.editor`), the role65 listed in the "IAM permissions" section of66 [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt),67 for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored68 events, and on-demand profiler sessions)69 - Monitoring Viewer (`roles/monitoring.viewer`) for the PromQL queries in Step70 271 - Logs Viewer (`roles/logging.viewer`) for the Cloud Logging query in Step 172 - Kubernetes Engine Viewer (`roles/container.viewer`) for the `kubectl get73 nodes` query in Step 374 - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin75 (`roles/container.clusterAdmin`)76- **Reference Documentation**:77 - [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)78 (Sections: "IAM permissions")79 - [Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt)80 (Sections: "Set up with gcloud CLI, Google Cloud console, or Terraform",81 "Manual installation", "Connection-operator")82 - [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)83 (Sections: "Get started", "Megascale XLA (MXLA) Hang Analyzer", "Access84 Workload Monitoring information through the API", "System Metrics")85 - [Get started with the ML Diagnostics CLI](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)86 (Sections: "List machine learning runs", "Monitored-events commands", "List87 profiler targets", "Capture on-demand profiler sessions")88 - [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)89 (Sections: "Before you contact support")90 - [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)91 (Sections: "Repair criteria", "Verify node auto-repair is enabled for a92 Standard node pool", "Enable auto-repair for an existing Standard node93 pool", "Node auto repair in TPU slice nodes")94 - [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)95 (Sections: "Configure auto repair for TPU slice nodes")96 - [Dump HLO Computations (OpenXLA)](https://openxla.org/xla/hlo_dumps)9798> **Read-only rule**: Run read-only diagnostic commands only. Never drain,99> delete, or re-create nodes, or run any other command that changes the cluster.100> Give the user any fix to apply themselves.101102When you recommend a fix, link the doc section that describes it.103104---105106## MXLA Hang Analyzer routing overview107108A Megascale hang occurs when a multi-slice worker has waited on a Megascale109communication operation for a set timeout period. The TPU logs then show a110Megascale `HANG_DETECTED` message. `HANG_DETECTED` is a catch-all signal that111the workload isn't progressing, and the cause can be in software or in hardware.112When a hang occurs, ML Diagnostics runs the `Megascale XLA (MXLA) Hang113Analyzer`, which reports the likely cause as a code. Consult the114[Megascale XLA (MXLA) Hang Analyzer](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)115section for the definition and recommended action of each code, and route by116category:117118- **Category A: compiler or HLO divergence (never cordon or replace nodes)**:119 - When the analyzer reports different HLO modules, inconsistent HLO120 compilation, or an inconsistent launch order across VMs (such as121 `FINGERPRINT_MISMATCH`), route to122 [Path A: Compiler or HLO divergence](references/path-a-compiler-hlo-divergence.md).123- **Category B: host program queueing or data input stall (never cordon or124 replace nodes)**:125 - When the analyzer reports that VMs aren't queuing programs to the TPU or126 that data input stalled (such as `DATA_INPUT_STALL`), route to127 [Path B: Host program queueing or data input stall](references/path-b-program-queueing-input-stall.md).128- **Category C: hardware or network faults on specific instances**:129 - When the analyzer attributes the hang to a TPU chip, SparseCore, ICI, or DCN130 networking issue, or to an unrecoverable error on specific instances, route131 to132 [Path C: Hardware or network faults](references/path-c-hardware-network-faults.md).133- **Category D: hang signal or event exists, but the analyzer is `NOT_DETECTED`134 or reports `UNKNOWN`**:135 - When `HANG_DETECTED` logs or a hang monitored event fired, but the analyzer136 report has `detectionState: "NOT_DETECTED"` or reports `UNKNOWN` ("The MXLA137 hang was detected but a potential cause is not determined"), route to138 [Path D: No DETECTED analyzer or UNKNOWN cause](references/path-d-no-detected-or-unknown.md).139140---141142## Diagnostic workflow143144### Step 0: Collect context and set the investigation window `[Low Risk]`145146Collect the target parameters. By default, query a 60-minute window `[T - 30m, T147+ 30m]` around `{issue_time}`:148149- `{project_id}`: Google Cloud project ID150- `{location}`: Google Cloud region where the ML run and GKE cluster reside (for151 example, `us-central1`)152- `{cluster_name}`: GKE cluster name153- `{namespace}` / `{workload_name}`: Kubernetes namespace and JobSet/Pod prefix154- `{ml_run_id}`: ML Diagnostics run ID155- `{issue_time}`: Timestamp when the hang occurred (`T`, ISO-8601 UTC)156- `{start_time}`: `T - 30m`157- `{end_time}`: `T + 30m`158159---160161### Step 1: Query `HANG_DETECTED` logs and hang events `[Low Risk]`1621631. **Check GKE container logs for `HANG_DETECTED` (read-only Cloud Logging164 LQL)**: Query `k8s_container` logs over `[{start_time}, {end_time}]` to165 confirm `HANG_DETECTED` and identify the first stalled pods:166167```sql168resource.type="k8s_container"169resource.labels.project_id="{project_id}"170resource.labels.cluster_name="{cluster_name}"171"HANG_DETECTED"172timestamp >= "{start_time}" AND timestamp <= "{end_time}"173```1741752. **List active or recent ML runs**: Follow the176 [List machine learning runs](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)177 section of the ML Diagnostics CLI reference (`gcloud alpha mldiagnostics178 machine-learning-run list`) to identify `{ml_run_id}`.1793. **List and describe hang monitored events**: Follow the180 [Monitored-events commands](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)181 section (`gcloud alpha mldiagnostics monitored-events list` and `gcloud alpha182 mldiagnostics monitored-events describe`) or183 [Access Workload Monitoring information through the API](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)184 to inspect the `Megascale XLA (MXLA) Hang Analyzer` report (`detectionState`,185 `details`, and `recommendedActions`).186 - If `HANG_DETECTED` logs or a hang monitored event fired, run Step 2 to187 correlate with the 1-minute metrics; if no analyzer reports188 `detectionState: "DETECTED"` or the analyzer reports `UNKNOWN`, follow189 [Path D: No DETECTED analyzer or UNKNOWN cause](references/path-d-no-detected-or-unknown.md).190 - If no `HANG_DETECTED` log or hang monitored event exists and multi-slice191 latencies are normal in Step 2, **rule out** an MXLA hang by following192 [Path E: Healthy telemetry](references/path-e-healthy-telemetry.md).193194---195196### Step 2: Correlate with 1-minute multi-slice and TPU metrics `[Low Risk]`197198Consult the199[System Metrics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)200section of the Workload Monitoring guide for the 1-minute multi-slice network201(`kubernetes.io/container/multislice/network/*`), multi-slice accelerator202(`kubernetes.io/container/multislice/accelerator/*`), and node duty-cycle203(`kubernetes.io/node/accelerator/duty_cycle`) metrics, and run read-only PromQL204queries over `[{start_time}, {end_time}]`:205206```promql207# 1. P95 multi-slice collective end-to-end latency by pod208histogram_quantile(209 0.95,210 sum by (pod_name, le) (211 rate(kubernetes_io:container_multislice_network_collective_end_to_end_latencies_bucket{212 monitored_resource="k8s_container",213 project_id="{project_id}",214 cluster_name="{cluster_name}"215 }[5m])216 )217)218219# 2. P95 multi-slice DCN transfer latency by pod220histogram_quantile(221 0.95,222 sum by (pod_name, le) (223 rate(kubernetes_io:container_multislice_network_dcn_transfer_latencies_bucket{224 monitored_resource="k8s_container",225 project_id="{project_id}",226 cluster_name="{cluster_name}"227 }[5m])228 )229)230231# 3. P95 host-to-device transfer latency by pod232histogram_quantile(233 0.95,234 sum by (pod_name, le) (235 rate(kubernetes_io:container_multislice_accelerator_host_to_device_transfer_latencies_bucket{236 monitored_resource="k8s_container",237 project_id="{project_id}",238 cluster_name="{cluster_name}"239 }[5m])240 )241)242243# 4. Node TPU duty cycle (Workload Monitoring detects a hang as a prolonged244# period of minimal to no TPU activity)245kubernetes_io:node_accelerator_duty_cycle{246 monitored_resource="k8s_node",247 project_id="{project_id}",248 cluster_name="{cluster_name}"249}250```251252---253254### Step 3: Map culprit Compute Engine instance IDs to GKE nodes `[Low Risk]`255256When the `Megascale XLA (MXLA) Hang Analyzer` reports culprit numeric Compute257Engine instance IDs in `details` or `recommendedActions`, map those numeric IDs258to GKE `Node` names and physical topology blocks using this read-only `kubectl`259query inspecting `container.googleapis.com/instance_id`:260261```bash262kubectl get nodes -l cloud.google.com/gke-tpu-accelerator \263 -o jsonpath='{range .items[*]}{.metadata.name}{"\tinstance_id="}{.metadata.annotations.container\.googleapis\.com/instance_id}{"\tblock="}{.metadata.labels.cloud\.google\.com/gce-topology-block}{"\tsubblock="}{.metadata.labels.cloud\.google\.com/gce-topology-subblock}{"\thost="}{.metadata.labels.cloud\.google\.com/gce-topology-host}{"\n"}{end}'264```265266---267268### Step 4: Remediation by root-cause category269270Load and follow only the reference file that matches the root-cause category271from Step 1:272273- **Path A (compiler or HLO divergence — different HLO modules, inconsistent HLO274 compilation, or inconsistent launch order across VMs, such as275 `FINGERPRINT_MISMATCH`)**: Read276 [Path A: Compiler or HLO divergence](references/path-a-compiler-hlo-divergence.md).277- **Path B (host program queueing or data input stall — `PROGRAM_NOT_QUEUED` or278 `DATA_INPUT_STALL`)**: Read279 [Path B: Host program queueing or data input stall](references/path-b-program-queueing-input-stall.md).280- **Path C (hardware, SparseCore, ICI, DCN networking faults, or281 `UNRECOVERABLE_ERROR` on specific instances)**: Read282 [Path C: Hardware or network faults](references/path-c-hardware-network-faults.md).283- **Path D (`HANG_DETECTED` logs or hang event fired, but the analyzer is284 `NOT_DETECTED` or reports `UNKNOWN`)**: Read285 [Path D: No DETECTED analyzer or UNKNOWN cause](references/path-d-no-detected-or-unknown.md).286- **Path E (no `HANG_DETECTED` logs or hang events, and steady telemetry)**:287 Read [Path E: Healthy telemetry](references/path-e-healthy-telemetry.md).288289---290291## Guardrails2922931. **Never change GKE-managed instance groups or VMs through Compute Engine**:294 Don't run `gcloud compute instance-groups managed` commands, such as295 `delete`, on a node pool's managed instance group. Handle nodes through GKE,296 as described in297 [Path C: Hardware or network faults](references/path-c-hardware-network-faults.md).2982. **Never cordon or replace nodes for compiler divergence or input stalls**: If299 the analyzer reports a compiler or HLO divergence, a program queueing issue,300 or a data input stall, don't cordon or replace TPU nodes. Follow301 [Path A: Compiler or HLO divergence](references/path-a-compiler-hlo-divergence.md)302 or303 [Path B: Host program queueing or data input stall](references/path-b-program-queueing-input-stall.md)304 instead.