Troubleshoot GKE TPU multi-slice hangs with the MXLA Hang Analyzer
Diagnose Cloud TPU multi-slice training hangs on Google Kubernetes Engine (GKE)
by correlating Megascale HANG_DETECTED logs and ML Diagnostics Workload
Monitoring Megascale XLA (MXLA) Hang Analyzer reports with 1-minute
multi-slice latency metrics (kubernetes.io/container/multislice/*) and GKE
node topology labels.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)"
and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang) to User-Agent headers.
Prerequisites
- Tools: Install the
Google Cloud SDK (
gcloud with
alpha component for gcloud alpha mldiagnostics) and kubectl.
- Cloud Billing & Project Configuration: Verify an active billing account is
linked (
gcloud billing projects describe {project_id}), authenticate
(gcloud auth login), set the target project (gcloud config set project {project_id}), and ensure container.googleapis.com,
logging.googleapis.com, monitoring.googleapis.com, and
hypercomputecluster.googleapis.com are enabled.
- Supported workloads and versions: Google Cloud ML Diagnostics only
supports JAX on TPUs (see
ML Diagnostics platform).
Workload Monitoring is enabled by default, supports the
jobset and job GKE
job types, and is compatible with GKE versions 1.36.0-gke.4681000 and later
(Configure GKE for ML Diagnostics,
which also covers the cluster setup needed for on-demand profiling in Step 4
Path B). If a workload uses another framework (such as PyTorch) or another
custom resource type, gcloud alpha mldiagnostics won't list ML runs or
monitored events for it. The Megascale XLA hang analyzer and Megascale XLA
metrics require LibTPU 0.40.0 or later (see "Get started" in
Workload monitoring with ML Diagnostics).
- Required IAM Roles:
- Cluster Director Editor (
roles/hypercomputecluster.editor), the role
listed in the "IAM permissions" section of
ML Diagnostics platform,
for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored
events, and on-demand profiler sessions)
- Monitoring Viewer (
roles/monitoring.viewer) for the PromQL queries in Step
2
- Logs Viewer (
roles/logging.viewer) for the Cloud Logging query in Step 1
- Kubernetes Engine Viewer (
roles/container.viewer) for the kubectl get nodes query in Step 3
- For remediation (
[High Risk] steps): Kubernetes Engine Cluster Admin
(roles/container.clusterAdmin)
- Reference Documentation:
- ML Diagnostics platform
(Sections: "IAM permissions")
- Configure GKE for ML Diagnostics
(Sections: "Set up with gcloud CLI, Google Cloud console, or Terraform",
"Manual installation", "Connection-operator")
- Workload monitoring with ML Diagnostics
(Sections: "Get started", "Megascale XLA (MXLA) Hang Analyzer", "Access
Workload Monitoring information through the API", "System Metrics")
- Get started with the ML Diagnostics CLI
(Sections: "List machine learning runs", "Monitored-events commands", "List
profiler targets", "Capture on-demand profiler sessions")
- Get support
(Sections: "Before you contact support")
- Auto-repair nodes
(Sections: "Repair criteria", "Verify node auto-repair is enabled for a
Standard node pool", "Enable auto-repair for an existing Standard node
pool", "Node auto repair in TPU slice nodes")
- Deploy TPU workloads in GKE Standard
(Sections: "Configure auto repair for TPU slice nodes")
- Dump HLO Computations (OpenXLA)
Read-only rule: Run read-only diagnostic commands only. Never drain,
delete, or re-create nodes, or run any other command that changes the cluster.
Give the user any fix to apply themselves.
When you recommend a fix, link the doc section that describes it.
MXLA Hang Analyzer routing overview
A Megascale hang occurs when a multi-slice worker has waited on a Megascale
communication operation for a set timeout period. The TPU logs then show a
Megascale HANG_DETECTED message. HANG_DETECTED is a catch-all signal that
the workload isn't progressing, and the cause can be in software or in hardware.
When a hang occurs, ML Diagnostics runs the Megascale XLA (MXLA) Hang Analyzer, which reports the likely cause as a code. Consult the
Megascale XLA (MXLA) Hang Analyzer
section for the definition and recommended action of each code, and route by
category:
- Category A: compiler or HLO divergence (never cordon or replace nodes):
- When the analyzer reports different HLO modules, inconsistent HLO
compilation, or an inconsistent launch order across VMs (such as
FINGERPRINT_MISMATCH), route to
Path A: Compiler or HLO divergence.
- Category B: host program queueing or data input stall (never cordon or
replace nodes):
- When the analyzer reports that VMs aren't queuing programs to the TPU or
that data input stalled (such as
DATA_INPUT_STALL), route to
Path B: Host program queueing or data input stall.
- Category C: hardware or network faults on specific instances:
- When the analyzer attributes the hang to a TPU chip, SparseCore, ICI, or DCN
networking issue, or to an unrecoverable error on specific instances, route
to
Path C: Hardware or network faults.
- Category D: hang signal or event exists, but the analyzer is
NOT_DETECTED
or reports UNKNOWN:
- When
HANG_DETECTED logs or a hang monitored event fired, but the analyzer
report has detectionState: "NOT_DETECTED" or reports UNKNOWN ("The MXLA
hang was detected but a potential cause is not determined"), route to
Path D: No DETECTED analyzer or UNKNOWN cause.
Diagnostic workflow
Step 0: Collect context and set the investigation window [Low Risk]
Collect the target parameters. By default, query a 60-minute window `[T - 30m, T
{project_id}: Google Cloud project ID
{location}: Google Cloud region where the ML run and GKE cluster reside (for
example, us-central1)
{cluster_name}: GKE cluster name
{namespace} / {workload_name}: Kubernetes namespace and JobSet/Pod prefix
{ml_run_id}: ML Diagnostics run ID
{issue_time}: Timestamp when the hang occurred (T, ISO-8601 UTC)
{start_time}: T - 30m
{end_time}: T + 30m
Step 1: Query HANG_DETECTED logs and hang events [Low Risk]
- Check GKE container logs for
HANG_DETECTED (read-only Cloud Logging
LQL): Query k8s_container logs over [{start_time}, {end_time}] to
confirm HANG_DETECTED and identify the first stalled pods:
resource.type="k8s_container"
resource.labels.project_id="{project_id}"
resource.labels.cluster_name="{cluster_name}"
"HANG_DETECTED"
timestamp >= "{start_time}" AND timestamp <= "{end_time}"
- List active or recent ML runs: Follow the
List machine learning runs
section of the ML Diagnostics CLI reference (
gcloud alpha mldiagnostics machine-learning-run list) to identify {ml_run_id}.
- List and describe hang monitored events: Follow the
Monitored-events commands
section (
gcloud alpha mldiagnostics monitored-events list and gcloud alpha mldiagnostics monitored-events describe) or
Access Workload Monitoring information through the API
to inspect the Megascale XLA (MXLA) Hang Analyzer report (detectionState,
details, and recommendedActions).
- If
HANG_DETECTED logs or a hang monitored event fired, run Step 2 to
correlate with the 1-minute metrics; if no analyzer reports
detectionState: "DETECTED" or the analyzer reports UNKNOWN, follow
Path D: No DETECTED analyzer or UNKNOWN cause.
- If no
HANG_DETECTED log or hang monitored event exists and multi-slice
latencies are normal in Step 2, rule out an MXLA hang by following
Path E: Healthy telemetry.
Step 2: Correlate with 1-minute multi-slice and TPU metrics [Low Risk]
Consult the
System Metrics
section of the Workload Monitoring guide for the 1-minute multi-slice network
(kubernetes.io/container/multislice/network/*), multi-slice accelerator
(kubernetes.io/container/multislice/accelerator/*), and node duty-cycle
(kubernetes.io/node/accelerator/duty_cycle) metrics, and run read-only PromQL
queries over [{start_time}, {end_time}]:
# 1. P95 multi-slice collective end-to-end latency by pod
histogram_quantile(
0.95,
sum by (pod_name, le) (
rate(kubernetes_io:container_multislice_network_collective_end_to_end_latencies_bucket{
monitored_resource="k8s_container",
project_id="{project_id}",
cluster_name="{cluster_name}"
}[5m])
)
)
# 2. P95 multi-slice DCN transfer latency by pod
histogram_quantile(
0.95,
sum by (pod_name, le) (
rate(kubernetes_io:container_multislice_network_dcn_transfer_latencies_bucket{
monitored_resource="k8s_container",
project_id="{project_id}",
cluster_name="{cluster_name}"
}[5m])
)
)
# 3. P95 host-to-device transfer latency by pod
histogram_quantile(
0.95,
sum by (pod_name, le) (
rate(kubernetes_io:container_multislice_accelerator_host_to_device_transfer_latencies_bucket{
monitored_resource="k8s_container",
project_id="{project_id}",
cluster_name="{cluster_name}"
}[5m])
)
)
# 4. Node TPU duty cycle (Workload Monitoring detects a hang as a prolonged
# period of minimal to no TPU activity)
kubernetes_io:node_accelerator_duty_cycle{
monitored_resource="k8s_node",
project_id="{project_id}",
cluster_name="{cluster_name}"
}
Step 3: Map culprit Compute Engine instance IDs to GKE nodes [Low Risk]
When the Megascale XLA (MXLA) Hang Analyzer reports culprit numeric Compute
Engine instance IDs in details or recommendedActions, map those numeric IDs
to GKE Node names and physical topology blocks using this read-only kubectl
query inspecting container.googleapis.com/instance_id:
kubectl get nodes -l cloud.google.com/gke-tpu-accelerator \
-o jsonpath='{range .items[*]}{.metadata.name}{"\tinstance_id="}{.metadata.annotations.container\.googleapis\.com/instance_id}{"\tblock="}{.metadata.labels.cloud\.google\.com/gce-topology-block}{"\tsubblock="}{.metadata.labels.cloud\.google\.com/gce-topology-subblock}{"\thost="}{.metadata.labels.cloud\.google\.com/gce-topology-host}{"\n"}{end}'
Step 4: Remediation by root-cause category
Load and follow only the reference file that matches the root-cause category
from Step 1:
- Path A (compiler or HLO divergence — different HLO modules, inconsistent HLO
compilation, or inconsistent launch order across VMs, such as
FINGERPRINT_MISMATCH): Read
Path A: Compiler or HLO divergence.
- Path B (host program queueing or data input stall —
PROGRAM_NOT_QUEUED or
DATA_INPUT_STALL): Read
Path B: Host program queueing or data input stall.
- Path C (hardware, SparseCore, ICI, DCN networking faults, or
UNRECOVERABLE_ERROR on specific instances): Read
Path C: Hardware or network faults.
- Path D (
HANG_DETECTED logs or hang event fired, but the analyzer is
NOT_DETECTED or reports UNKNOWN): Read
Path D: No DETECTED analyzer or UNKNOWN cause.
- Path E (no
HANG_DETECTED logs or hang events, and steady telemetry):
Read Path E: Healthy telemetry.
Guardrails
- Never change GKE-managed instance groups or VMs through Compute Engine:
Don't run
gcloud compute instance-groups managed commands, such as
delete, on a node pool's managed instance group. Handle nodes through GKE,
as described in
Path C: Hardware or network faults.
- Never cordon or replace nodes for compiler divergence or input stalls: If
the analyzer reports a compiler or HLO divergence, a program queueing issue,
or a data input stall, don't cordon or replace TPU nodes. Follow
Path A: Compiler or HLO divergence
or
Path B: Host program queueing or data input stall
instead.
1---2description: Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze without progressing steps, emit `HANG_DETECTED`, or stall in collective operations. Don't use for gradual step-time throughput drops without hangs (use gke-ai-troubleshooting-tpu-performance-degradation) or pod preemption/eviction restarts (use gke-ai-troubleshooting-jobset-interruption).3metadata:4 source: https://skillmd.com/skills/h3y6e/gke-ai-troubleshooting-tpu-mxla-hang5 category: AiAndMachineLearning6 github-path: skills/cloud/gke-ai-troubleshooting-tpu-mxla-hang7 github-ref: refs/heads/main8 github-repo: https://github.com/google/skills9 github-tree-sha: ee7386ee81dcdb3cc8e69d89e746e5a76a02bddd10 version: 1.0.111name: gke-ai-troubleshooting-tpu-mxla-hang12---1314# Troubleshoot GKE TPU multi-slice hangs with the MXLA Hang Analyzer1516Diagnose Cloud TPU multi-slice training hangs on Google Kubernetes Engine (GKE)17by correlating Megascale `HANG_DETECTED` logs and **ML Diagnostics Workload18Monitoring** `Megascale XLA (MXLA) Hang Analyzer` reports with **1-minute19multi-slice latency metrics** (`kubernetes.io/container/multislice/*`) and GKE20node topology labels.2122---2324## Attribution2526**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you27MUST read [Usage Attribution](references/attribution.md). For commands executed28directly in the terminal (never in generated scripts, Terraform, or user code),29prefix `gcloud` with30`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)"`31and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)` to `User-Agent` headers.3233## Prerequisites3435- **Tools**: Install the36 [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) (`gcloud` with37 `alpha` component for `gcloud alpha mldiagnostics`) and `kubectl`.38- **Cloud Billing & Project Configuration**: Verify an active billing account is39 linked (`gcloud billing projects describe {project_id}`), authenticate40 (`gcloud auth login`), set the target project (`gcloud config set project41 {project_id}`), and ensure `container.googleapis.com`,42 `logging.googleapis.com`, `monitoring.googleapis.com`, and43 `hypercomputecluster.googleapis.com` are enabled.44- **Supported workloads and versions**: Google Cloud ML Diagnostics only45 supports JAX on TPUs (see46 [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)).47 Workload Monitoring is enabled by default, supports the `jobset` and `job` GKE48 job types, and is compatible with GKE versions `1.36.0-gke.4681000` and later49 ([Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt),50 which also covers the cluster setup needed for on-demand profiling in Step 451 Path B). If a workload uses another framework (such as PyTorch) or another52 custom resource type, `gcloud alpha mldiagnostics` won't list ML runs or53 monitored events for it. The Megascale XLA hang analyzer and Megascale XLA54 metrics require LibTPU `0.40.0` or later (see "Get started" in55 [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)).56- **Required IAM Roles**:57 - Cluster Director Editor (`roles/hypercomputecluster.editor`), the role58 listed in the "IAM permissions" section of59 [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt),60 for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored61 events, and on-demand profiler sessions)62 - Monitoring Viewer (`roles/monitoring.viewer`) for the PromQL queries in Step63 264 - Logs Viewer (`roles/logging.viewer`) for the Cloud Logging query in Step 165 - Kubernetes Engine Viewer (`roles/container.viewer`) for the `kubectl get66 nodes` query in Step 367 - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin68 (`roles/container.clusterAdmin`)69- **Reference Documentation**:70 - [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)71 (Sections: "IAM permissions")72 - [Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt)73 (Sections: "Set up with gcloud CLI, Google Cloud console, or Terraform",74 "Manual installation", "Connection-operator")75 - [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)76 (Sections: "Get started", "Megascale XLA (MXLA) Hang Analyzer", "Access77 Workload Monitoring information through the API", "System Metrics")78 - [Get started with the ML Diagnostics CLI](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)79 (Sections: "List machine learning runs", "Monitored-events commands", "List80 profiler targets", "Capture on-demand profiler sessions")81 - [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)82 (Sections: "Before you contact support")83 - [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)84 (Sections: "Repair criteria", "Verify node auto-repair is enabled for a85 Standard node pool", "Enable auto-repair for an existing Standard node86 pool", "Node auto repair in TPU slice nodes")87 - [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)88 (Sections: "Configure auto repair for TPU slice nodes")89 - [Dump HLO Computations (OpenXLA)](https://openxla.org/xla/hlo_dumps)9091> **Read-only rule**: Run read-only diagnostic commands only. Never drain,92> delete, or re-create nodes, or run any other command that changes the cluster.93> Give the user any fix to apply themselves.9495When you recommend a fix, link the doc section that describes it.9697---9899## MXLA Hang Analyzer routing overview100101A Megascale hang occurs when a multi-slice worker has waited on a Megascale102communication operation for a set timeout period. The TPU logs then show a103Megascale `HANG_DETECTED` message. `HANG_DETECTED` is a catch-all signal that104the workload isn't progressing, and the cause can be in software or in hardware.105When a hang occurs, ML Diagnostics runs the `Megascale XLA (MXLA) Hang106Analyzer`, which reports the likely cause as a code. Consult the107[Megascale XLA (MXLA) Hang Analyzer](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)108section for the definition and recommended action of each code, and route by109category:110111- **Category A: compiler or HLO divergence (never cordon or replace nodes)**:112 - When the analyzer reports different HLO modules, inconsistent HLO113 compilation, or an inconsistent launch order across VMs (such as114 `FINGERPRINT_MISMATCH`), route to115 [Path A: Compiler or HLO divergence](references/path-a-compiler-hlo-divergence.md).116- **Category B: host program queueing or data input stall (never cordon or117 replace nodes)**:118 - When the analyzer reports that VMs aren't queuing programs to the TPU or119 that data input stalled (such as `DATA_INPUT_STALL`), route to120 [Path B: Host program queueing or data input stall](references/path-b-program-queueing-input-stall.md).121- **Category C: hardware or network faults on specific instances**:122 - When the analyzer attributes the hang to a TPU chip, SparseCore, ICI, or DCN123 networking issue, or to an unrecoverable error on specific instances, route124 to125 [Path C: Hardware or network faults](references/path-c-hardware-network-faults.md).126- **Category D: hang signal or event exists, but the analyzer is `NOT_DETECTED`127 or reports `UNKNOWN`**:128 - When `HANG_DETECTED` logs or a hang monitored event fired, but the analyzer129 report has `detectionState: "NOT_DETECTED"` or reports `UNKNOWN` ("The MXLA130 hang was detected but a potential cause is not determined"), route to131 [Path D: No DETECTED analyzer or UNKNOWN cause](references/path-d-no-detected-or-unknown.md).132133---134135## Diagnostic workflow136137### Step 0: Collect context and set the investigation window `[Low Risk]`138139Collect the target parameters. By default, query a 60-minute window `[T - 30m, T140+ 30m]` around `{issue_time}`:141142- `{project_id}`: Google Cloud project ID143- `{location}`: Google Cloud region where the ML run and GKE cluster reside (for144 example, `us-central1`)145- `{cluster_name}`: GKE cluster name146- `{namespace}` / `{workload_name}`: Kubernetes namespace and JobSet/Pod prefix147- `{ml_run_id}`: ML Diagnostics run ID148- `{issue_time}`: Timestamp when the hang occurred (`T`, ISO-8601 UTC)149- `{start_time}`: `T - 30m`150- `{end_time}`: `T + 30m`151152---153154### Step 1: Query `HANG_DETECTED` logs and hang events `[Low Risk]`1551561. **Check GKE container logs for `HANG_DETECTED` (read-only Cloud Logging157 LQL)**: Query `k8s_container` logs over `[{start_time}, {end_time}]` to158 confirm `HANG_DETECTED` and identify the first stalled pods:159160```sql161resource.type="k8s_container"162resource.labels.project_id="{project_id}"163resource.labels.cluster_name="{cluster_name}"164"HANG_DETECTED"165timestamp >= "{start_time}" AND timestamp <= "{end_time}"166```1671682. **List active or recent ML runs**: Follow the169 [List machine learning runs](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)170 section of the ML Diagnostics CLI reference (`gcloud alpha mldiagnostics171 machine-learning-run list`) to identify `{ml_run_id}`.1723. **List and describe hang monitored events**: Follow the173 [Monitored-events commands](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)174 section (`gcloud alpha mldiagnostics monitored-events list` and `gcloud alpha175 mldiagnostics monitored-events describe`) or176 [Access Workload Monitoring information through the API](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)177 to inspect the `Megascale XLA (MXLA) Hang Analyzer` report (`detectionState`,178 `details`, and `recommendedActions`).179 - If `HANG_DETECTED` logs or a hang monitored event fired, run Step 2 to180 correlate with the 1-minute metrics; if no analyzer reports181 `detectionState: "DETECTED"` or the analyzer reports `UNKNOWN`, follow182 [Path D: No DETECTED analyzer or UNKNOWN cause](references/path-d-no-detected-or-unknown.md).183 - If no `HANG_DETECTED` log or hang monitored event exists and multi-slice184 latencies are normal in Step 2, **rule out** an MXLA hang by following185 [Path E: Healthy telemetry](references/path-e-healthy-telemetry.md).186187---188189### Step 2: Correlate with 1-minute multi-slice and TPU metrics `[Low Risk]`190191Consult the192[System Metrics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)193section of the Workload Monitoring guide for the 1-minute multi-slice network194(`kubernetes.io/container/multislice/network/*`), multi-slice accelerator195(`kubernetes.io/container/multislice/accelerator/*`), and node duty-cycle196(`kubernetes.io/node/accelerator/duty_cycle`) metrics, and run read-only PromQL197queries over `[{start_time}, {end_time}]`:198199```promql200# 1. P95 multi-slice collective end-to-end latency by pod201histogram_quantile(202 0.95,203 sum by (pod_name, le) (204 rate(kubernetes_io:container_multislice_network_collective_end_to_end_latencies_bucket{205 monitored_resource="k8s_container",206 project_id="{project_id}",207 cluster_name="{cluster_name}"208 }[5m])209 )210)211212# 2. P95 multi-slice DCN transfer latency by pod213histogram_quantile(214 0.95,215 sum by (pod_name, le) (216 rate(kubernetes_io:container_multislice_network_dcn_transfer_latencies_bucket{217 monitored_resource="k8s_container",218 project_id="{project_id}",219 cluster_name="{cluster_name}"220 }[5m])221 )222)223224# 3. P95 host-to-device transfer latency by pod225histogram_quantile(226 0.95,227 sum by (pod_name, le) (228 rate(kubernetes_io:container_multislice_accelerator_host_to_device_transfer_latencies_bucket{229 monitored_resource="k8s_container",230 project_id="{project_id}",231 cluster_name="{cluster_name}"232 }[5m])233 )234)235236# 4. Node TPU duty cycle (Workload Monitoring detects a hang as a prolonged237# period of minimal to no TPU activity)238kubernetes_io:node_accelerator_duty_cycle{239 monitored_resource="k8s_node",240 project_id="{project_id}",241 cluster_name="{cluster_name}"242}243```244245---246247### Step 3: Map culprit Compute Engine instance IDs to GKE nodes `[Low Risk]`248249When the `Megascale XLA (MXLA) Hang Analyzer` reports culprit numeric Compute250Engine instance IDs in `details` or `recommendedActions`, map those numeric IDs251to GKE `Node` names and physical topology blocks using this read-only `kubectl`252query inspecting `container.googleapis.com/instance_id`:253254```bash255kubectl get nodes -l cloud.google.com/gke-tpu-accelerator \256 -o jsonpath='{range .items[*]}{.metadata.name}{"\tinstance_id="}{.metadata.annotations.container\.googleapis\.com/instance_id}{"\tblock="}{.metadata.labels.cloud\.google\.com/gce-topology-block}{"\tsubblock="}{.metadata.labels.cloud\.google\.com/gce-topology-subblock}{"\thost="}{.metadata.labels.cloud\.google\.com/gce-topology-host}{"\n"}{end}'257```258259---260261### Step 4: Remediation by root-cause category262263Load and follow only the reference file that matches the root-cause category264from Step 1:265266- **Path A (compiler or HLO divergence — different HLO modules, inconsistent HLO267 compilation, or inconsistent launch order across VMs, such as268 `FINGERPRINT_MISMATCH`)**: Read269 [Path A: Compiler or HLO divergence](references/path-a-compiler-hlo-divergence.md).270- **Path B (host program queueing or data input stall — `PROGRAM_NOT_QUEUED` or271 `DATA_INPUT_STALL`)**: Read272 [Path B: Host program queueing or data input stall](references/path-b-program-queueing-input-stall.md).273- **Path C (hardware, SparseCore, ICI, DCN networking faults, or274 `UNRECOVERABLE_ERROR` on specific instances)**: Read275 [Path C: Hardware or network faults](references/path-c-hardware-network-faults.md).276- **Path D (`HANG_DETECTED` logs or hang event fired, but the analyzer is277 `NOT_DETECTED` or reports `UNKNOWN`)**: Read278 [Path D: No DETECTED analyzer or UNKNOWN cause](references/path-d-no-detected-or-unknown.md).279- **Path E (no `HANG_DETECTED` logs or hang events, and steady telemetry)**:280 Read [Path E: Healthy telemetry](references/path-e-healthy-telemetry.md).281282---283284## Guardrails2852861. **Never change GKE-managed instance groups or VMs through Compute Engine**:287 Don't run `gcloud compute instance-groups managed` commands, such as288 `delete`, on a node pool's managed instance group. Handle nodes through GKE,289 as described in290 [Path C: Hardware or network faults](references/path-c-hardware-network-faults.md).2912. **Never cordon or replace nodes for compiler divergence or input stalls**: If292 the analyzer reports a compiler or HLO divergence, a program queueing issue,293 or a data input stall, don't cordon or replace TPU nodes. Follow294 [Path A: Compiler or HLO divergence](references/path-a-compiler-hlo-divergence.md)295 or296 [Path B: Host program queueing or data input stall](references/path-b-program-queueing-input-stall.md)297 instead.