GKE JobSet Interruption Troubleshooting
Use this skill to systematically diagnose and resolve JobSet interruptions,
restarts, and preemptions on GKE clusters hosting large-scale AI/ML workloads.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-jobset-interruption)"
and append agent-skills/1.0 (skill:gke-ai-troubleshooting-jobset-interruption) to User-Agent headers.
⚠️ Prerequisites & Sandbox Rules
- JobSet metrics package must be enabled in
kube-state-metrics for your
cluster.
- Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.
- Autonomous & Sandbox Execution Rule: If API queries, tools, or commands
encounter
403 Permission Denied, authentication errors, or network
isolation, do NOT enter authentication or credential troubleshooting
loops. Populate the query templates with the acquired variables
({project_id}, {cluster_name}, {workload_name}, {start_time},
{end_time}), inspect any locally staged telemetry or mock data files if
available, and complete the diagnostic workflow and resolution
recommendations autonomously.
🔍 Diagnostic Workflow
Step 0: Context Acquisition & Time Window Definition
Independently gather context using tools, workspace files, environment details,
or user prompt context:
- Project ID (
{project_id})
- Cluster Name (
{cluster_name})
- Workload Name (JobSet Name) (
{workload_name})
- Workload Namespace (
{namespace})
- Issue Time (
{issue_time})
If specific variables are not explicitly provided by the user, inspect cluster
resources or logs to determine them, or use the {variable} placeholders
provided.
Time Handling Rules
- Autonomous Time Window: If a relative time (e.g., "X minutes ago") or no
exact timestamp is provided, calculate the query window based on current
time or available log timestamps.
- Window Calculation: If a timestamp
{issue_time} is available (or
calculated as T), set {start_time} = T - 30m and {end_time} = T + 30m.
Step 1: Identify JobSet Restarts and Attempts [Low Risk]
Verify if the JobSet is experiencing restart loops and determine the frequency
of restarts.
Visual Chart / MQL Query - restarts
PromQL Metric Query - restarts
PromQL Query Specification:
kube_jobset_restarts{jobset_name="{workload_name}", cluster="{cluster_name}"}
Diagnostic Logic: A non-zero or increasing value for restarts indicates
that the JobSet is being actively restarted by the controller due to worker
failure or interruption.
Automation: Proceed to Step 2 automatically after reporting findings.
Step 2: Inspect Nodepool Interruptions [Low Risk]
Determine if the JobSet restarts were triggered by physical nodepool-level
events (such as spot preemptions, maintenance, or host terminations).
A. Metrics Query (Nodepool Interruption Counts)
Visual Chart / MQL Query - interruptions
MQL Query Specification:
fetch k8s_node_pool
| metric 'kubernetes.io/node_pool/interruption_count'
| filter cluster_name == '{cluster_name}'
| align next_older(10m)
| every 10m
| group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]
PromQL Query - interruptions
PromQL Query Specification:
sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (
avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name="{cluster_name}"}[10m])
)
B. Log Query (Nodepool Life Events)
LQL Log Filter Specification:
resource.type="gke_nodepool"
AND resource.labels.cluster_name="{cluster_name}"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"
Diagnostic Logic:
- PreemptionEvent: Spot VMs were preempted, or node was scale-down.
- MaintenanceEvent: Node pool updated or Google scheduled maintenance.
- TerminationEvent: Serious host failures. Check
interruption_reason
or logs for host issues.
- See Failure Signatures for examples
of node termination logs and preemption events.
Automation: Proceed to Step 3 automatically.
Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]
Correlate node readiness failures with physical host VMs to see if a single
faulty host repeatedly fails coordinator pods.
A. Metrics Query (Node Ready Status Check)
Visual Chart / MQL Query - node status
MQL Query Specification:
fetch k8s_node
| metric 'kubernetes.io/node/status_condition'
| filter cluster_name == '{cluster_name}' && metric.condition == 'Ready' && metric.status == 'False'
| align next_older(1m)
| every 1m
| group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]
PromQL Query - node status
PromQL Query Specification:
sum by (status, condition, node_pool_name) (
kubernetes_io:node_status_condition{cluster_name="{cluster_name}", condition="Ready", status="False"}
)
B. Metrics Query (Node-to-Host Metadata Topology Correlation)
MQL Query Specification:
fetch k8s_node
| metric 'kubernetes.io/node/cpu/total_cores'
| filter cluster_name == '{cluster_name}'
| align next_older(1m)
| every 1m
| group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]
C. Log Query (Node Fault Logs)
LQL Log Filter Specification:
resource.type="k8s_node"
AND resource.labels.cluster_name="{cluster_name}"
AND (textPayload:"host error" OR textPayload:"kernel panic" OR textPayload:"hardware failure" OR textPayload:"NodeNotReady")
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"
Diagnostic Logic: Identify if specific nodes are unhealthy
(Ready=False or Unknown) and correlate them to their GCE physical host
ID via metadata.user.gce_topology_host. Check if the same host is
repeatedly failing.
Automation: Proceed to Step 4 automatically.
Step 4: Inspect Pod and Worker / Container Failures [Low Risk]
Analyze pod status phases and retrieve coordinator worker logs to identify
application-level crashes or network deadlocks.
Required Execution Order: You MUST analyze pod status phases (Section A)
and unschedulable pod metrics (Section B) to assess overall workload health
before inspecting specific worker container logs (Section C).
A. Metrics Query (Pod Lifecycle Phases)
Visual Chart / MQL Query - pod phase
PromQL Query - pod phase
PromQL Query Specification:
sum by (phase) (
avg_over_time(kube_pod_status_phase{cluster="{cluster_name}", pod=~"{workload_name}.*"}[10m])
)
B. Metrics Query (Unschedulable Pod Count)
C. Log Query (Worker Container Logs)
LQL Log Filter Specification:
resource.type="k8s_container"
AND resource.labels.cluster_name="{cluster_name}"
AND labels."k8s-pod/jobset_sigs_k8s_io/jobset-name"="{workload_name}"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"
Diagnostic Logic:
- Check the pod timeline to spot pending or unschedulable pods.
- Use worker container logs to analyze worker 0 in slice 0 (coordinator)
for NCCL timeouts, collective communication issues, or MegaScale hangs.
Automation: Proceed to Resolution.
🛠️ Resolution Workflow
Resolution 1: Preemption & Autoscaling Optimizations [Low Risk]
If Step 2 showed high preemption counts on Spot VMs:
- Action: Suggest switching critical long-running training workloads to
GKE Reserved/On-Demand VMs or utilizing Compact Placement Policies
to minimize defragmentation interruptions.
- Justification: Eliminates spot-market preemptions and reduces training
restarts.
Resolution 2: Quarantine Faulty Host VMs [High Risk]
If Step 3 identified a specific host ID (gce-topology-host) that consistently
fails or triggers restarts across multiple attempts:
- Action: Recommend cordoning/draining the GKE node, deleting the
underlying GCE VM instance to trigger instance recreation, and opening a
support ticket with Google Cloud Support specifying the physical host ID.
- Justification: GKE auto-repair will recreate the VM instance on healthy
physical hardware, preventing infinite restart loops.
📋 Copypaste Checklist
1---2description: Diagnoses GKE JobSet interruptions, restarts, and preemptions for AI/ML training workloads autonomously. Use when troubleshooting JobSet restart loops, spot VM preemptions, node readiness failures, host VM issues, or coordinator worker crashes. Don't use for general GKE cluster creation, basic workload deployment, or non-JobSet application issues.3metadata:4 source: https://skillmd.com/skills/h3y6e/gke-ai-troubleshooting-jobset-interruption5 category: Containers6 github-path: skills/cloud/gke-ai-troubleshooting-jobset-interruption7 github-ref: refs/heads/main8 github-repo: https://github.com/google/skills9 github-tree-sha: adfe352d67a8c6f034f9c70f222c3602108f892b10 version: 1.0.111name: gke-ai-troubleshooting-jobset-interruption12---1314# GKE JobSet Interruption Troubleshooting1516Use this skill to systematically diagnose and resolve JobSet interruptions,17restarts, and preemptions on GKE clusters hosting large-scale AI/ML workloads.1819## Attribution2021**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you22MUST read [Usage Attribution](references/attribution.md). For commands executed23directly in the terminal (never in generated scripts, Terraform, or user code),24prefix `gcloud` with25`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-jobset-interruption)"`26and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-jobset-interruption)` to `User-Agent` headers.2728## ⚠️ Prerequisites & Sandbox Rules2930- JobSet metrics package must be enabled in `kube-state-metrics` for your31 cluster.32- Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.33- **Autonomous & Sandbox Execution Rule**: If API queries, tools, or commands34 encounter `403 Permission Denied`, authentication errors, or network35 isolation, **do NOT enter authentication or credential troubleshooting36 loops**. Populate the query templates with the acquired variables37 (`{project_id}`, `{cluster_name}`, `{workload_name}`, `{start_time}`,38 `{end_time}`), inspect any locally staged telemetry or mock data files if39 available, and complete the diagnostic workflow and resolution40 recommendations autonomously.4142## 🔍 Diagnostic Workflow4344### Step 0: Context Acquisition & Time Window Definition4546Independently gather context using tools, workspace files, environment details,47or user prompt context:4849- **Project ID** (`{project_id}`)50- **Cluster Name** (`{cluster_name}`)51- **Workload Name (JobSet Name)** (`{workload_name}`)52- **Workload Namespace** (`{namespace}`)53- **Issue Time** (`{issue_time}`)5455If specific variables are not explicitly provided by the user, inspect cluster56resources or logs to determine them, or use the `{variable}` placeholders57provided.5859#### Time Handling Rules60611. **Autonomous Time Window**: If a relative time (e.g., "X minutes ago") or no62 exact timestamp is provided, calculate the query window based on current63 time or available log timestamps.642. **Window Calculation**: If a timestamp `{issue_time}` is available (or65 calculated as `T`), set `{start_time}` = `T - 30m` and `{end_time}` = `T +66 30m`.6768--------------------------------------------------------------------------------6970### Step 1: Identify JobSet Restarts and Attempts [Low Risk]7172Verify if the JobSet is experiencing restart loops and determine the frequency73of restarts.7475#### Visual Chart / MQL Query - restarts7677- **MQL Query Specification**:7879 ```mql80 fetch prometheus_target81 | metric 'prometheus.googleapis.com/kube_jobset_restarts/gauge'82 | filter resource.cluster_name == '{cluster_name}' && metric.jobset_name == '{workload_name}'83 | align next_older(1m)84 | every 1m85 | group_by [metric.jobset_name], [val: max(value)]86 ```8788#### PromQL Metric Query - restarts8990- **PromQL Query Specification**:9192 ```promql93 kube_jobset_restarts{jobset_name="{workload_name}", cluster="{cluster_name}"}94 ```9596- **Diagnostic Logic**: A non-zero or increasing value for restarts indicates97 that the JobSet is being actively restarted by the controller due to worker98 failure or interruption.99100- **Automation**: Proceed to Step 2 automatically after reporting findings.101102--------------------------------------------------------------------------------103104### Step 2: Inspect Nodepool Interruptions [Low Risk]105106Determine if the JobSet restarts were triggered by physical nodepool-level107events (such as spot preemptions, maintenance, or host terminations).108109#### A. Metrics Query (Nodepool Interruption Counts)110111##### Visual Chart / MQL Query - interruptions112113- **MQL Query Specification**:114115 ```mql116 fetch k8s_node_pool117 | metric 'kubernetes.io/node_pool/interruption_count'118 | filter cluster_name == '{cluster_name}'119 | align next_older(10m)120 | every 10m121 | group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]122 ```123124##### PromQL Query - interruptions125126- **PromQL Query Specification**:127128 ```promql129 sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (130 avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name="{cluster_name}"}[10m])131 )132 ```133134#### B. Log Query (Nodepool Life Events)135136- **LQL Log Filter Specification**:137138 ```sql139 resource.type="gke_nodepool"140 AND resource.labels.cluster_name="{cluster_name}"141 AND timestamp >= "{start_time}"142 AND timestamp <= "{end_time}"143 ```144145- **Diagnostic Logic**:146147 - **PreemptionEvent**: Spot VMs were preempted, or node was scale-down.148 - **MaintenanceEvent**: Node pool updated or Google scheduled maintenance.149 - **TerminationEvent**: Serious host failures. Check `interruption_reason`150 or logs for host issues.151 - See [Failure Signatures](references/failure_signatures.md) for examples152 of node termination logs and preemption events.153154- **Automation**: Proceed to Step 3 automatically.155156--------------------------------------------------------------------------------157158### Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]159160Correlate node readiness failures with physical host VMs to see if a single161faulty host repeatedly fails coordinator pods.162163#### A. Metrics Query (Node Ready Status Check)164165##### Visual Chart / MQL Query - node status166167- **MQL Query Specification**:168169 ```mql170 fetch k8s_node171 | metric 'kubernetes.io/node/status_condition'172 | filter cluster_name == '{cluster_name}' && metric.condition == 'Ready' && metric.status == 'False'173 | align next_older(1m)174 | every 1m175 | group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]176 ```177178##### PromQL Query - node status179180- **PromQL Query Specification**:181182 ```promql183 sum by (status, condition, node_pool_name) (184 kubernetes_io:node_status_condition{cluster_name="{cluster_name}", condition="Ready", status="False"}185 )186 ```187188#### B. Metrics Query (Node-to-Host Metadata Topology Correlation)189190- **MQL Query Specification**:191192 ```mql193 fetch k8s_node194 | metric 'kubernetes.io/node/cpu/total_cores'195 | filter cluster_name == '{cluster_name}'196 | align next_older(1m)197 | every 1m198 | group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]199 ```200201#### C. Log Query (Node Fault Logs)202203- **LQL Log Filter Specification**:204205 ```sql206 resource.type="k8s_node"207 AND resource.labels.cluster_name="{cluster_name}"208 AND (textPayload:"host error" OR textPayload:"kernel panic" OR textPayload:"hardware failure" OR textPayload:"NodeNotReady")209 AND timestamp >= "{start_time}"210 AND timestamp <= "{end_time}"211 ```212213- **Diagnostic Logic**: Identify if specific nodes are unhealthy214 (`Ready=False` or `Unknown`) and correlate them to their GCE physical host215 ID via `metadata.user.gce_topology_host`. Check if the same host is216 repeatedly failing.217218- **Automation**: Proceed to Step 4 automatically.219220--------------------------------------------------------------------------------221222### Step 4: Inspect Pod and Worker / Container Failures [Low Risk]223224Analyze pod status phases and retrieve coordinator worker logs to identify225application-level crashes or network deadlocks.226227> **Required Execution Order**: You MUST analyze pod status phases (Section A)228> and unschedulable pod metrics (Section B) to assess overall workload health229> before inspecting specific worker container logs (Section C).230231#### A. Metrics Query (Pod Lifecycle Phases)232233##### Visual Chart / MQL Query - pod phase234235- **MQL Query Specification**:236237 ```mql238 fetch k8s_pod239 | metric 'kubernetes.io/pod/status/phase'240 | filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'241 | align next_older(10m)242 | every 10m243 | group_by [metric.phase], [val: count()]244 ```245246##### PromQL Query - pod phase247248- **PromQL Query Specification**:249250 ```promql251 sum by (phase) (252 avg_over_time(kube_pod_status_phase{cluster="{cluster_name}", pod=~"{workload_name}.*"}[10m])253 )254 ```255256#### B. Metrics Query (Unschedulable Pod Count)257258- **MQL Query Specification**:259260 ```mql261 fetch k8s_pod262 | metric 'kubernetes.io/pod/status/unschedulable'263 | filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'264 | align next_older(10m)265 | every 10m266 | group_by [pod_name], [val: max(value)]267 ```268269#### C. Log Query (Worker Container Logs)270271- **LQL Log Filter Specification**:272273 ```sql274 resource.type="k8s_container"275 AND resource.labels.cluster_name="{cluster_name}"276 AND labels."k8s-pod/jobset_sigs_k8s_io/jobset-name"="{workload_name}"277 AND timestamp >= "{start_time}"278 AND timestamp <= "{end_time}"279 ```280281- **Diagnostic Logic**:282283 1. Check the pod timeline to spot pending or unschedulable pods.284 2. Use worker container logs to analyze worker 0 in slice 0 (coordinator)285 for NCCL timeouts, collective communication issues, or MegaScale hangs.286287- **Automation**: Proceed to Resolution.288289--------------------------------------------------------------------------------290291## 🛠️ Resolution Workflow292293### Resolution 1: Preemption & Autoscaling Optimizations [Low Risk]294295If Step 2 showed high preemption counts on Spot VMs:296297- **Action**: Suggest switching critical long-running training workloads to298 **GKE Reserved/On-Demand VMs** or utilizing **Compact Placement Policies**299 to minimize defragmentation interruptions.300- **Justification**: Eliminates spot-market preemptions and reduces training301 restarts.302303### Resolution 2: Quarantine Faulty Host VMs [High Risk]304305If Step 3 identified a specific host ID (`gce-topology-host`) that consistently306fails or triggers restarts across multiple attempts:307308- **Action**: Recommend cordoning/draining the GKE node, deleting the309 underlying GCE VM instance to trigger instance recreation, and opening a310 support ticket with Google Cloud Support specifying the physical host ID.311- **Justification**: GKE auto-repair will recreate the VM instance on healthy312 physical hardware, preventing infinite restart loops.313314--------------------------------------------------------------------------------315316## 📋 Copypaste Checklist317318- [ ] Gather context and compute `{start_time}` (`{issue_time} - 30m`) and319 `{end_time}` (`{issue_time} + 30m`) window.320- [ ] Query JobSet restart attempts.321- [ ] Check Nodepool interruptions (spot preemptions vs. hardware322 terminations).323- [ ] Query node-to-host mapping and check node logs for physical host errors.324- [ ] Inspect pod timeline status and coordinator worker container logs.325- [ ] Recommend appropriate scheduling strategy (On-demand vs Spot) or host VM326 quarantining.