GKE Kueue Workload & Admission Debugging Playbook
[!WARNING] This skill is experimental and under active validation for Cluster Toolkit. Diagnostic queries are read-only, but remediation plans must be reviewed carefully by cluster operators before execution.
Use this playbook when distributed AI/ML training jobs (PyTorchJob, RayCluster, JobSet, Job, LeaderWorkerSet) submitted to GKE remain indefinitely in Pending, Inadmissible, Admitted: False, or experience unexpected evictions.
1. Diagnostic Triage (Read-Only)
Step 0: Fast-Path Workload Inspection (SRE First Response)
If investigating a parent Job, JobSet, or RayCluster, locate its generated Workload name first:
# Method A: Direct lookup via owner UID (safely queries installed controllers without failing on missing CRDs)
JOB_UID=$(kubectl get job <PARENT_JOB_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.uid}' 2>/dev/null)
if [ -z "$JOB_UID" ]; then
for kind in jobset raycluster rayjob pytorchjob mpijob leaderworkerset; do
JOB_UID=$(kubectl get "$kind" <PARENT_JOB_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.uid}' 2>/dev/null)
[ -n "$JOB_UID" ] && break
done
fi
[ -n "$JOB_UID" ] && kubectl get workloads -n <NAMESPACE> -l "kueue.x-k8s.io/job-uid=$JOB_UID"
# Method B: Universal filter by parent job name prefix
kubectl get workloads -n <NAMESPACE> | grep <PARENT_JOB_NAME>
Always inspect high-level workload events and conditions:
kubectl describe workload <WORKLOAD_NAME> -n <NAMESPACE>
Look directly at the bottom Events: section for QuotaReserved, AdmissionCheckFailed, Inadmissible, Preempted, or WorkloadEvicted.
Step 1: Query Workload Conditions, Evictions & LocalQueue Assignment
Inspect workloads in the target namespace. Do NOT query Pods directly -- Kueue suspends Jobs before Pod creation (.spec.suspend: true).
kubectl get workloads.kueue.x-k8s.io -n <NAMESPACE> -o custom-columns='NAME:.metadata.name,QUEUE:.spec.queueName,RESERVED:.status.conditions[?(@.type=="QuotaReserved")].status,ADMITTED:.status.conditions[?(@.type=="Admitted")].status,EVICTED:.status.conditions[?(@.type=="Evicted")].status,RESERVED_REASON:.status.conditions[?(@.type=="QuotaReserved")].reason,EVICTED_REASON:.status.conditions[?(@.type=="Evicted")].reason,MSG:.status.conditions[?(@.type=="QuotaReserved")].message'
Condition Interpretation:
QuotaReserved: False,Reason: PendingorInadmissible: Workload is either waiting for quota (Step 3) OR failed flavor matching (Step 4). Always inspectMSG:- If
MSGindicatesinsufficient quota ... > maximum capacity, proceed to Step 3. - If
MSGindicatesdoesn't match node affinityoruntolerated taint, proceed to Step 4 and Step 6.
- If
QuotaReserved: True,Admitted: False: Quota is reserved, but the workload is gated by AdmissionChecks (e.g., GKE DWS Flex-start, TPU Dynamic Slicing). Inspect checks:kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{range .status.admissionChecks[*]}{"Check: "}{.name}{" | State: "}{.state}{" | RequeueAfter: "}{.requeueAfterSeconds}{"s | Message: "}{.message}{"\n"}{end}'Evicted: True: Workload was interrupted after admission. CheckEVICTED_REASON:Preempted: Evicted to reclaim quota or satisfy higher-priority job. Proceed to Step 5.PodsReadyTimeout: Worker pods failed to become ready withinwaitForPodsReady.timeout(check pod image pulls / driver daemonsets).AdmissionCheck: An admission check transitioned to Retry/Rejected while running.Deactivation: Workload deactivated (spec.active: false) due to a rejected admission check.ClusterQueueStopped/LocalQueueStopped: Queue was paused withstopPolicy: HoldAndDrain.
Step 2: Verify LocalQueue & Upstream ClusterQueue Binding
Verify that the LocalQueue exists, routes to an active ClusterQueue, and is not administratively stopped:
kubectl get localqueue <QUEUE_NAME> -n <NAMESPACE> -o custom-columns='NAME:.metadata.name,CLUSTER_QUEUE:.spec.clusterQueue,STOP_POLICY:.spec.stopPolicy,ACTIVE:.status.conditions[?(@.type=="Active")].status,REASON:.status.conditions[?(@.type=="Active")].reason,PENDING:.status.pendingWorkloads,RESERVING:.status.reservingWorkloads,ADMITTED:.status.admittedWorkloads'
Root Cause Check:
- If
STOP_POLICYisHoldorHoldAndDrain, the queue is inactive and will not admit new workloads. - If
ACTIVEis False or the resource does not exist, the workload will remain pending indefinitely. RESERVINGindicates workloads that have passed quota checks but are awaiting AdmissionChecks.
Step 3: Audit ClusterQueue Quotas, Cohorts, and Reservation vs. Usage
Query the upstream ClusterQueue without dumping full YAML. Extract queue policy, nominal quotas, borrowing limits, reservations, and active usage:
# 1. Inspect ClusterQueue policy, cohort, and strategy
kubectl get clusterqueue <CLUSTER_QUEUE_NAME> -o jsonpath='{"ClusterQueue: "}{.metadata.name}{"\nCohort: "}{.spec.cohortName}{.spec.cohort}{"\nQueueingStrategy: "}{.spec.queueingStrategy}{"\nStopPolicy: "}{.spec.stopPolicy}{"\nPendingWorkloads: "}{.status.pendingWorkloads}{" | ReservingWorkloads: "}{.status.reservingWorkloads}{" | AdmittedWorkloads: "}{.status.admittedWorkloads}{"\n---\n"}{range .spec.resourceGroups[*].flavors[*]}{"Flavor: "}{.name}{"\n"}{range .resources[*]}{" Resource: "}{.name}{" | Nominal: "}{.nominalQuota}{" | BorrowingLimit: "}{.borrowingLimit}{"\n"}{end}{end}'
# 2. Check live reservations vs. actual running pod usage
kubectl get clusterqueue <CLUSTER_QUEUE_NAME> -o jsonpath='{"=== FLAVORS RESERVATION (Admitted + Reserving) ===\n"}{range .status.flavorsReservation[*]}{"Flavor: "}{.name}{"\n"}{range .resources[*]}{" Resource: "}{.name}{" | Reserved: "}{.total}{" | Borrowed: "}{.borrowed}{"\n"}{end}{end}{"=== FLAVORS USAGE (Actively Admitted Pods) ===\n"}{range .status.flavorsUsage[*]}{"Flavor: "}{.name}{"\n"}{range .resources[*]}{" Resource: "}{.name}{" | Used: "}{.total}{" | Borrowed: "}{.borrowed}{"\n"}{end}{end}'
Root Cause Check:
- Head-of-Line (HOL) Blocking: If
QueueingStrategyisStrictFIFO, an inadmissible older job at the head of the queue blocks all newer jobs across the cluster even if quota is free! Inspect oldest pending workloads for this queue:# In target namespace (sorted by creation timestamp to spot head-of-line blockers) kubectl get workloads -n <NAMESPACE> --sort-by=.metadata.creationTimestamp -o custom-columns='NAME:.metadata.name,QUEUE:.spec.queueName,RESERVED:.status.conditions[?(@.type=="QuotaReserved")].status,CREATED:.metadata.creationTimestamp' | head -n 10 # Or across all namespaces sharing this ClusterQueue kubectl get workloads -A --sort-by=.metadata.creationTimestamp -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,QUEUE:.spec.queueName,RESERVED:.status.conditions[?(@.type=="QuotaReserved")].status,CREATED:.metadata.creationTimestamp' | head -n 10 - Cohort Saturation: If
borrowingLimitis empty/null, borrowing is unlimited up to the cohort's total available quota. Iftotal >= nominalQuotaand cohort borrowing is exhausted, the queue must wait for active jobs to finish. For multi-tenant quota sharing architectures, refer to the GKE Kueue Cohort Tutorial. - Reservation vs. Usage Delta: If
Reservedis high butUsedis 0, quota is tied up by workloads waiting onAdmissionChecks(e.g. DWS capacity acquisition).
[!NOTE] Tenant RBAC Fallback: If querying
ClusterQueuereturnsError from server (Forbidden), inspect the namespaced admission message directly:kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{.status.conditions[*].message}'
Step 4: Verify ResourceFlavor Node Selectors, Taints, and Nodepools
When QuotaReserved: False with Reason: Pending or Inadmissible (and MSG indicates "couldn't assign flavors" or "doesn't match node affinity"):
# 1. Inspect ResourceFlavors in cluster (nodeLabels, nodeTaints, tolerations, topology)
kubectl get resourceflavor -o jsonpath='{range .items[*]}{"Flavor: "}{.metadata.name}{"\nNodeLabels: "}{.spec.nodeLabels}{"\nNodeTaints: "}{.spec.nodeTaints}{"\nTolerations: "}{.spec.tolerations}{"\nTopologyName: "}{.spec.topologyName}{"\n\n"}{end}'
# Or inspect a single flavor directly:
# kubectl get resourceflavor <FLAVOR_NAME> -o jsonpath='{"Flavor: "}{.metadata.name}{"\nNodeLabels: "}{.spec.nodeLabels}{"\nNodeTaints: "}{.spec.nodeTaints}{"\nTolerations: "}{.spec.tolerations}{"\nTopologyName: "}{.spec.topologyName}{"\n"}'
# 2. GKE accelerator node inspection (custom columns for GPU/TPU labels, taints, and nodepools)
kubectl get nodes -o custom-columns='NAME:.metadata.name,NODEPOOL:.metadata.labels.cloud\.google\.com/gke-nodepool,GPU:.metadata.labels.cloud\.google\.com/gke-accelerator,TPU:.metadata.labels.cloud\.google\.com/gke-tpu-accelerator,TPU_TOPOLOGY:.metadata.labels.cloud\.google\.com/gke-tpu-topology,TAINTS:.spec.taints' | head -n 25
# For large clusters, target accelerator pools directly with label selectors:
# kubectl get nodes -l 'cloud.google.com/gke-accelerator' -o custom-columns=... | head -n 25
# kubectl get nodes -l 'cloud.google.com/gke-tpu-accelerator' -o custom-columns=... | head -n 25
# 3. Inspect requested accelerator resource types in the workload's PodSets
kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{range .spec.podSets[*]}{"PodSet: "}{.name}{" | Count: "}{.count}{" | Requests: "}{.template.spec.containers[*].resources.requests}{"\n"}{end}'
Verify that the workload's Pod template (or individual spec.podSets[*]):
- Heterogeneous / Gang Scheduling: For multi-podSet workloads (e.g., RayCluster with CPU head and GPU workers, or LeaderWorkerSet), check the
pod set <name>prefix inMSGto identify which specific podSet failed flavor assignment.
- NVIDIA GPU Jobs (
nvidia.com/gpu): Has toleration for GPU node taints usingoperator: Exists(e.g.,key: nvidia.com/gpu, operator: Exists, effect: NoSchedule) and nodeSelector matchingcloud.google.com/gke-accelerator(e.g.,nvidia-h100-80gb,nvidia-h100-mega-80gb,nvidia-l4). If running on GKE Dynamic Workload Scheduler (DWS Flex) node pools, also verify toleration forcloud.google.com/gke-queued=true:NoSchedule. - Google TPU Jobs (
google.com/tpu): Has toleration for TPU node taints usingoperator: Exists(e.g.,key: google.com/tpu, operator: Exists, effect: NoSchedule). Verify nodeSelector matchescloud.google.com/gke-tpu-accelerator(e.g.tpu-v5p-slice,tpu-v6e-slice) and topology matchescloud.google.com/gke-tpu-topology(e.g.2x2x1for 3D torus,2x4for 2D mesh). If running on GKE DWS Flex node pools, also verify toleration forcloud.google.com/gke-queued=true:NoSchedule. - Has
nodeSelector/nodeAffinitycompatible with the flavor'snodeLabels.
Step 5: Audit Workload Priority, Flavor Fungibility & Preemption
If a high-priority workload is blocked behind lower-priority workloads, inspect priority classes, fungibility, and preemption policies:
# 1. Check Workload Priority and WorkloadPriorityClass
kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{"PriorityClass: "}{.spec.priorityClassName}{" | Priority: "}{.spec.priority}{"\n"}'
# 2. Check ClusterQueue Preemption and Flavor Fungibility
kubectl get clusterqueue <CLUSTER_QUEUE_NAME> -o jsonpath='{"Preemption: "}{.spec.preemption}{"\nFlavorFungibility: "}{.spec.flavorFungibility}{"\n"}'
Preemption Check:
- If
withinClusterQueueisNever, high-priority workloads cannot preempt lower-priority workloads in the same queue. - If
reclaimWithinCohortisNever, workloads cannot reclaim borrowed quota from peer ClusterQueues. - If
FlavorFungibility.whenCanPreemptisTryNextFlavor, Kueue attempts to find available quota in fallback flavors before preempting running workloads.
Step 6: Topology-Aware Scheduling (TAS) & AdmissionCheck Deep-Dive
A. Diagnosing Topology-Aware Scheduling (TAS) Failures
When multi-node distributed training jobs specify compact placement:
# 1. Check if the workload requested a specific topology (TopologyRequest or PodSet annotations)
kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{range .spec.podSets[*]}{"PodSet: "}{.name}{" | TopologyRequest: "}{.topologyRequest}{" | Annotations: "}{.template.metadata.annotations}{"\n"}{end}'
# 2. Inspect Topology hierarchy and levels
kubectl get topology -o custom-columns='NAME:.metadata.name,LEVELS:.spec.levels'
If nodes are fragmented across racks/blocks, TAS cannot satisfy contiguous block requirements, leaving the job Inadmissible.
B. Diagnosing GKE Dynamic Workload Scheduler (DWS Flex) Provisioning
When using DWS Flex-start (ProvisioningRequest):
kubectl get provisioningrequests.autoscaling.x-k8s.io -n <NAMESPACE> -o custom-columns='NAME:.metadata.name,PROVISIONED:.status.conditions[?(@.type=="Provisioned")].status,FAILED:.status.conditions[?(@.type=="Failed")].status,BOOKED:.status.conditions[?(@.type=="BookingExpired")].status'
If FAILED is True, DWS rejected the reservation due to regional stockout. The workload will enter AdmissionCheck: Retry or Rejected.
2. Safety Guidelines & Blast Radius Protocol
- Safe Commands (Zero Turn Delay): Read-only commands (
kubectl get,kubectl describe,kubectl logs) execute freely during diagnosis. - Mutating Commands (Confirmation Strictly Required): Deleting workloads, evicting pods, restarting controllers, altering ClusterQueue quota, or changing stop policies must NEVER execute without presenting a structured remediation plan and obtaining user confirmation.
Required Remediation Plan Format:
[PROPOSED REMEDIATION PLAN]
- Target Resource: <ClusterQueue|LocalQueue|Workload>/<RESOURCE_NAME>
- Root Cause Identified: <Precise diagnostic finding>
- Proposed Action: <Exact command to be executed>
- Blast Radius: <Affected workloads, namespaces, and cohort tenants>
- Confirmation Required: Reply 'yes' to proceed.
Canonical Non-Destructive Patch Recipes:
- Unpausing a LocalQueue (
stopPolicy: None):kubectl patch localqueue <NAME> -n <NAMESPACE> --type=merge -p '{"spec":{"stopPolicy":"None"}}' - Switching StrictFIFO to BestEffortFIFO (resolving Head-of-Line deadlock):
kubectl patch clusterqueue <NAME> --type=merge -p '{"spec":{"queueingStrategy":"BestEffortFIFO"}}' - Reactivating an Inactive Workload (
spec.active: true):kubectl patch workload <NAME> -n <NAMESPACE> --type=merge -p '{"spec":{"active":true}}'
3. References & Canonical Documentation
- Kueue Concepts & Architecture:
references/kueue-docs.md| kueue.sigs.k8s.io/docs/concepts - GKE Cohorts & DWS Hardware Profiles:
references/gke-kueue-docs.md| docs.cloud.google.com/kubernetes-engine/docs/tutorials/kueue-cohort