Troubleshoot TPU slice creation failures caused by partition overlap
A dynamic slice fails to form with the status reason SliceCreationFailed when,
among other documented causes, any of its nodes are already used by another
slice. This skill confirms the failure, rules out the other documented causes,
finds the slice that uses the same nodes, and routes the fix through GKE.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)"
and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap) to User-Agent headers.
Prerequisites
- Tools:
- Cloud Billing & Project Configuration: Ensure the project has an active
Cloud Billing account. Follow
Before you begin
to initialize
gcloud and enable the Google Kubernetes Engine API
(container.googleapis.com), ensure the Cloud Monitoring API
(monitoring.googleapis.com) is enabled, and follow
Store cluster information for kubectl
to configure kubectl cluster credentials.
- Required IAM Roles:
- Kubernetes Engine Viewer (
roles/container.viewer)
- Monitoring Viewer (
roles/monitoring.viewer)
- Documentation:
- Use dynamic slicing with a custom scheduler
(Sections: "Before you begin", "Target your node pool to belong to a block
or a sub-block", "Verify the status of the nodes and the partitions",
"Create a Slice custom resource", "Monitor the status of the Slice custom
resource", "Delete the slice")
- Configure cluster access for kubectl
(Sections: "Install kubectl", "Install required plugins", "Store cluster
information for kubectl")
- Slice
(Sections: "Slice status", "Slice specification")
- Schedule dynamic slices with Kueue and TAS
(Sections: "Target your node pool to belong to a block or a sub-block",
"Create a dynamic slice with Kueue and TAS", "Define the partition health
selection", "Optimize workload scheduling with slice reuse", "Enable Slice
reuse", "Enforce slice deletion per workload", "Monitor the status of the
slice", "Monitor the metrics of the slice", "Clean up")
- GKE system metrics
(Sections: "Kubernetes metrics")
- Monitored resource types
(Sections: "k8s_node_pool")
- About GKE dynamic slicing
(Sections: "Dynamic slicing configurations")
- Get support
(Sections: "Before you contact support", "Recommendations for efficient
support")
Read-only rule: Run read-only diagnostic commands only. Never drain,
delete, or re-create nodes, or run any other command that changes the cluster.
Give the user any fix to apply themselves.
When you recommend a fix, link the doc section that describes it.
Diagnostic workflow
Step 0: Collect context [Low Risk]
Collect the target parameters:
{project_id}: Google Cloud project ID
{cluster_name}: GKE cluster name
{location}: Cluster region or zone
{slice_name}: Name of the Slice custom resource that failed to form
{node_pool_name}: Name of the TPU node pool (or node pools) backing the
partitions in {slice_name}, if known. If the user doesn't know
{node_pool_name}, after Step 1 run the Cloud Monitoring query in Step 3
for the partition IDs in {partition_ids} and read the node_pool_name
label on the
k8s_node_pool
monitored resource.
{issue_time}: Incident timestamp in RFC3339 UTC, if known. If the user
doesn't know {issue_time}, use the Last Transition Time
(lastTransitionTime, "A timestamp of the most recent status change" in
Slice status)
of the Ready condition from Step 1. By default, query Cloud Monitoring over
a 60-minute window around {issue_time}.
Step 1: Confirm the SliceCreationFailed status [Low Risk]
To describe {slice_name}, follow the instructions in the section
Monitor the status of the Slice custom resource.
- In
Status > Conditions, check that the Reason of the Ready condition
is SliceCreationFailed, and record its Last Transition Time (to use as
{issue_time} if needed).
- Record the IDs in
Spec > Partition Ids as {partition_ids}, the
Topology, and any Labels and Annotations on {slice_name}.
- If the
Ready reason isn't SliceCreationFailed, this skill doesn't apply.
See
Slice status
for what each reason means, and hand off to
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring.
Step 2: Rule out the other documented causes [Low Risk]
Per
Create a Slice custom resource,
slice creation also fails with SliceCreationFailed when:
- The selected nodes don't exist.
- The topology doesn't match the number of partitions. A dynamic sub-slice lists
exactly one partition ID. A dynamic super-slice lists
(A × B × C) / 64
partition IDs.
If either of those causes applies, hand off to
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring.
Otherwise, follow
Verify the status of the nodes and the partitions
on {node_pool_name}:
- Verify that the nodes exist and carry each ID in
{partition_ids}.
- On the nodes that carry
{partition_ids}, record every partition ID of
every shape (4x4x4, 2x4x4, 2x2x4, 2x2x2, and 2x2x1) from the
cloud.google.com/gke-tpu-partition-.*-id labels. Per
Dynamic slicing configurations,
dynamic sub-slicing partitions a 4x4x4 sub-block into smaller topologies on
the same physical node pool, so partitions of different shapes share the same
nodes.
- Read each partition's
cloud.google.com/gke-tpu-partition-.*-state label. Per
Verify the status of the nodes and the partitions
and
Define the partition health selection,
count HEALTHY partitions and DEGRADED 4x4x4 partitions ("still usable
for dynamic slice formation"; "Smaller topologies don't have a degraded
state") as available. Partitions in UNHEALTHY ("can't be used to form a
slice"), UNSET, or INCOMPLETE aren't available. If a requested partition
is in UNHEALTHY, UNSET, or INCOMPLETE, suggest an available partition
whose nodes no other slice uses (Step 4, item 2), hand off to
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring, or escalate to Cloud
Customer Care (Step 4, item 4).
Step 3: Find the slice that uses the same nodes [Low Risk]
Per
Create a Slice custom resource,
slice creation fails with SliceCreationFailed when any nodes on the custom
resource are used by another slice. Because partitions of different shapes on
the same sub-block share nodes (see
Dynamic slicing configurations),
another slice conflicts with {slice_name} if it lists any of the partition
IDs (of any shape) recorded on {slice_name}'s nodes in Step 2.
- List the
Slice custom resources in the cluster (as shown in
Delete the slice)
and describe each one following
Monitor the status of the Slice custom resource
to read Spec > Partition Ids, Metadata > Labels, and Status >
Conditions (Ready reason):
- Treat any slice other than
{slice_name} whose Spec > Partition Ids
includes any partition ID recorded on {slice_name}'s nodes in Step 2
(whether the exact ID in {partition_ids} or another shape's partition ID
on the same nodes) as conflicting. Record its name, its Ready reason
(such as ACTIVE), and whether Kueue manages it (Step 4, item 1).
- To check partition-to-slice associations in Cloud Monitoring (including
around
{issue_time}), query the kubernetes.io/accelerator/partition/state
metric listed in
Monitor the metrics of the slice.
Per
Kubernetes metrics,
this metric "is 1 if the partition state is detected, 0 otherwise", and its
labels include partition_id, slice_id ("ID of the associated slice"),
and state ("Health of the partition (e.g., HEALTHY, UNHEALTHY)"). Run the
query for each {partition_id} recorded on {slice_name}'s nodes in Step 2:
kubernetes_io:accelerator_partition_state{
monitored_resource="k8s_node_pool",
cluster_name="{cluster_name}",
partition_id="{partition_id}"
}
- Next steps based on Step 3 results:
- Another slice uses the nodes now: Proceed to Step 4.
- No other slice uses the nodes now, but the metric (value
1) showed an
associated slice_id around {issue_time}: The conflict may already be
over. Proceed to Step 4, item 1 (to re-create or retry {slice_name}) and
then Step 4, item 4.
- Neither the current
Slice custom resources nor the metric around
{issue_time} shows another slice: Node overlap isn't confirmed. Hand
off to gke-ai-troubleshooting-tpu-dynamic-slices-monitoring or escalate
to Cloud Customer Care (Step 4, item 4).
Step 4: Resolution [High Risk]
Guardrails:
- Never delete or change the other slice, or stop its workload, to free
the nodes. Deleting a
Slice custom resource starts dismantling that slice
(DEACTIVATING, see
Monitor the status of the Slice custom resource).
The owner of that slice decides when to release it.
- Never manually modify or delete
Slice custom resources created by
Kueue. Per
Target your node pool to belong to a block or a sub-block,
"Avoid manually modifying Slice custom resources created by Kueue."
- Never change GKE-managed instance groups or VMs directly through Compute
Engine, and never delete node pools to free a partition. Manage node pools
and slices through GKE.
First, check whether Kueue manages {slice_name} or the conflicting slice:
- Per
Monitor the status of the slice,
when a slice is created or actively used by a workload, the slice controller
attaches the
kueue.gke.io/tpu-slice-used-by-namespace and
kueue.gke.io/tpu-slice-used-by-workload labels.
- Per
Optimize workload scheduling with slice reuse
and
Enable Slice reuse,
when slice reuse is enabled, "Kueue marks the slice as idle at the end of a
job instead of deleting it", and "Reusable slices are named dynamically with
an
s-<topology> prefix, decoupling them from specific workload names."
Then apply the matching option:
- Wait for the conflicting slice to release the nodes and retry:
- If Kueue manages the conflicting slice (it has the
kueue.gke.io/...
labels or an s-<topology> name prefix): Do not suggest deleting the
Slice custom resource directly. Point the workload owner to
Clean up
(deleting the JobSet triggers Kueue to delete the associated Slice
custom resources) and, if slice reuse keeps an idle s-<topology> slice
allocated after a job ends, to
Enforce slice deletion per workload
(kueue.gke.io/tpu-enforce-slice-deletion: "true").
- If a custom scheduler or user created the conflicting slice directly:
Its owner can delete it when finished, as described in
Delete the slice.
- How
{slice_name} retries: Check whether {slice_name} already has
slice.gke.io/retry-on-failure: "true" in Metadata > Annotations. Per
Create a Slice custom resource,
when a Slice custom resource is created with this annotation, "If the
slice isn't created because of the SliceCreationFailed status reason, the
controller will retry until the slice is successfully formed." Because the
documentation only shows configuring
slice.gke.io/retry-on-failure: "true" in the Slice manifest when
creating a Slice custom resource (and doesn't state that adding the
annotation to an existing slice already in SliceCreationFailed starts a
retry), if {slice_name} wasn't created by Kueue and lacks the annotation,
tell the user to delete {slice_name} following
Delete the slice
and re-create it with slice.gke.io/retry-on-failure: "true" in its
manifest following
Create a Slice custom resource.
- Use available partitions whose nodes no other slice uses: If the user
manages
Slice custom resources directly (not through Kueue) and can't wait,
direct them to
Verify the status of the nodes and the partitions
and
Create a Slice custom resource
so they can pick available partitions (HEALTHY partitions, or DEGRADED
4x4x4 partitions) whose nodes aren't used by any other slice (across all
partition shapes on those nodes) and create the Slice custom resource with
them.
- Schedule dynamic slices with Kueue and TAS instead of creating
Slice
custom resources directly: Per
Target your node pool to belong to a block or a sub-block
and
Create a dynamic slice with Kueue and TAS,
instead of creating Slice custom resources directly, teams can schedule
GKE workloads with Kueue and Topology Aware Scheduling (TAS), where "Kueue
automatically handles the creation and deletion of Slice custom resources"
and admits a workload after its slice becomes active.
- Verify or escalate: Repeat Step 1 following
Monitor the status of the Slice custom resource:
ACTIVE: The slice is fully formed, healthy, and ready to execute
workloads.
ACTIVE_DEGRADED: The slice is formed, and workloads can run, but
performance might be impacted.
- Still
SliceCreationFailed after no other slice uses the nodes, or
FAILED: Hand off to
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring, or advise the user
to contact Cloud Customer Care following
Before you contact support
and
Recommendations for efficient support.
1---2name: gke-ai-troubleshooting-tpu-partition-overlap3description: >-4 Diagnose and resolve GKE TPU slice creation failures (SliceCreationFailed)5 caused by partition overlap conflicts where requested TPU partitions or their6 nodes are already used by another slice or JobSet. Use when diagnosing GKE TPU7 slice creation failures with SliceCreationFailed partition or node conflicts.8 Don't use for general slice lifecycle states or slice provisioning failures9 without node overlap (use10 gke-ai-troubleshooting-tpu-dynamic-slices-monitoring), quota exhaustion, or11 unresponsive TPU nodes.12metadata:13 source: https://skillmd.com/skills/google/gke-ai-troubleshooting-tpu-partition-overlap14 version: "1.0.1"15 category: Containers16---1718# Troubleshoot TPU slice creation failures caused by partition overlap1920A dynamic slice fails to form with the status reason `SliceCreationFailed` when,21among other documented causes, any of its nodes are already used by another22slice. This skill confirms the failure, rules out the other documented causes,23finds the slice that uses the same nodes, and routes the fix through GKE.2425## Attribution2627**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you28MUST read [Usage Attribution](references/attribution.md). For commands executed29directly in the terminal (never in generated scripts, Terraform, or user code),30prefix `gcloud` with31`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)"`32and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)` to `User-Agent` headers.3334## Prerequisites3536- **Tools**:37 - For installing and initializing `gcloud` (which covers authentication and38 project selection), follow39 [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).40 - For installing `kubectl` and required plugins, follow41 [Install kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)42 and43 [Install required plugins](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt).44- **Cloud Billing & Project Configuration**: Ensure the project has an active45 Cloud Billing account. Follow46 [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)47 to initialize `gcloud` and enable the Google Kubernetes Engine API48 (`container.googleapis.com`), ensure the Cloud Monitoring API49 (`monitoring.googleapis.com`) is enabled, and follow50 [Store cluster information for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)51 to configure `kubectl` cluster credentials.52- **Required IAM Roles**:53 - Kubernetes Engine Viewer (`roles/container.viewer`)54 - Monitoring Viewer (`roles/monitoring.viewer`)55- **Documentation**:56 - [Use dynamic slicing with a custom scheduler](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)57 (Sections: "Before you begin", "Target your node pool to belong to a block58 or a sub-block", "Verify the status of the nodes and the partitions",59 "Create a Slice custom resource", "Monitor the status of the Slice custom60 resource", "Delete the slice")61 - [Configure cluster access for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)62 (Sections: "Install kubectl", "Install required plugins", "Store cluster63 information for kubectl")64 - [Slice](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/slice.md.txt)65 (Sections: "Slice status", "Slice specification")66 - [Schedule dynamic slices with Kueue and TAS](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)67 (Sections: "Target your node pool to belong to a block or a sub-block",68 "Create a dynamic slice with Kueue and TAS", "Define the partition health69 selection", "Optimize workload scheduling with slice reuse", "Enable Slice70 reuse", "Enforce slice deletion per workload", "Monitor the status of the71 slice", "Monitor the metrics of the slice", "Clean up")72 - [GKE system metrics](https://docs.cloud.google.com/monitoring/api/metrics_kubernetes.md.txt)73 (Sections: "Kubernetes metrics")74 - [Monitored resource types](https://docs.cloud.google.com/monitoring/api/resources.md.txt)75 (Sections: "k8s_node_pool")76 - [About GKE dynamic slicing](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt)77 (Sections: "Dynamic slicing configurations")78 - [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)79 (Sections: "Before you contact support", "Recommendations for efficient80 support")8182> **Read-only rule**: Run read-only diagnostic commands only. Never drain,83> delete, or re-create nodes, or run any other command that changes the cluster.84> Give the user any fix to apply themselves.8586When you recommend a fix, link the doc section that describes it.8788---8990## Diagnostic workflow9192### Step 0: Collect context `[Low Risk]`9394Collect the target parameters:9596- `{project_id}`: Google Cloud project ID97- `{cluster_name}`: GKE cluster name98- `{location}`: Cluster region or zone99- `{slice_name}`: Name of the Slice custom resource that failed to form100- `{node_pool_name}`: Name of the TPU node pool (or node pools) backing the101 partitions in `{slice_name}`, if known. If the user doesn't know102 `{node_pool_name}`, after Step 1 run the Cloud Monitoring query in Step 3103 for the partition IDs in `{partition_ids}` and read the `node_pool_name`104 label on the105 [k8s_node_pool](https://docs.cloud.google.com/monitoring/api/resources.md.txt)106 monitored resource.107- `{issue_time}`: Incident timestamp in RFC3339 UTC, if known. If the user108 doesn't know `{issue_time}`, use the `Last Transition Time`109 (`lastTransitionTime`, "A timestamp of the most recent status change" in110 [Slice status](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/slice.md.txt))111 of the `Ready` condition from Step 1. By default, query Cloud Monitoring over112 a 60-minute window around `{issue_time}`.113114---115116### Step 1: Confirm the `SliceCreationFailed` status `[Low Risk]`117118To describe `{slice_name}`, follow the instructions in the section119[Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).120121- In `Status` > `Conditions`, check that the `Reason` of the `Ready` condition122 is `SliceCreationFailed`, and record its `Last Transition Time` (to use as123 `{issue_time}` if needed).124- Record the IDs in `Spec` > `Partition Ids` as `{partition_ids}`, the125 `Topology`, and any `Labels` and `Annotations` on `{slice_name}`.126- If the `Ready` reason isn't `SliceCreationFailed`, this skill doesn't apply.127 See128 [Slice status](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/slice.md.txt)129 for what each reason means, and hand off to130 `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.131132---133134### Step 2: Rule out the other documented causes `[Low Risk]`135136Per137[Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),138slice creation also fails with `SliceCreationFailed` when:139140- The selected nodes don't exist.141- The topology doesn't match the number of partitions. A dynamic sub-slice lists142 exactly one partition ID. A dynamic super-slice lists `(A × B × C) / 64`143 partition IDs.144145If either of those causes applies, hand off to146`gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.147148Otherwise, follow149[Verify the status of the nodes and the partitions](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)150on `{node_pool_name}`:151152- Verify that the nodes exist and carry each ID in `{partition_ids}`.153- On the nodes that carry `{partition_ids}`, record **every** partition ID of154 **every** shape (`4x4x4`, `2x4x4`, `2x2x4`, `2x2x2`, and `2x2x1`) from the155 `cloud.google.com/gke-tpu-partition-.*-id` labels. Per156 [Dynamic slicing configurations](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt),157 dynamic sub-slicing partitions a `4x4x4` sub-block into smaller topologies on158 the same physical node pool, so partitions of different shapes share the same159 nodes.160- Read each partition's `cloud.google.com/gke-tpu-partition-.*-state` label. Per161 [Verify the status of the nodes and the partitions](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)162 and163 [Define the partition health selection](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt),164 count `HEALTHY` partitions and `DEGRADED` `4x4x4` partitions ("still usable165 for dynamic slice formation"; "Smaller topologies don't have a degraded166 state") as available. Partitions in `UNHEALTHY` ("can't be used to form a167 slice"), `UNSET`, or `INCOMPLETE` aren't available. If a requested partition168 is in `UNHEALTHY`, `UNSET`, or `INCOMPLETE`, suggest an available partition169 whose nodes no other slice uses (Step 4, item 2), hand off to170 `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`, or escalate to Cloud171 Customer Care (Step 4, item 4).172173---174175### Step 3: Find the slice that uses the same nodes `[Low Risk]`176177Per178[Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),179slice creation fails with `SliceCreationFailed` when any nodes on the custom180resource are used by another slice. Because partitions of different shapes on181the same sub-block share nodes (see182[Dynamic slicing configurations](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt)),183another slice conflicts with `{slice_name}` if it lists **any** of the partition184IDs (of any shape) recorded on `{slice_name}`'s nodes in Step 2.1851861. List the `Slice` custom resources in the cluster (as shown in187 [Delete the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt))188 and describe each one following189 [Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)190 to read `Spec` > `Partition Ids`, `Metadata` > `Labels`, and `Status` >191 `Conditions` (`Ready` reason):192 - Treat any slice other than `{slice_name}` whose `Spec` > `Partition Ids`193 includes **any** partition ID recorded on `{slice_name}`'s nodes in Step 2194 (whether the exact ID in `{partition_ids}` or another shape's partition ID195 on the same nodes) as conflicting. Record its name, its `Ready` reason196 (such as `ACTIVE`), and whether Kueue manages it (Step 4, item 1).1972. To check partition-to-slice associations in Cloud Monitoring (including198 around `{issue_time}`), query the `kubernetes.io/accelerator/partition/state`199 metric listed in200 [Monitor the metrics of the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt).201 Per202 [Kubernetes metrics](https://docs.cloud.google.com/monitoring/api/metrics_kubernetes.md.txt),203 this metric "is 1 if the partition state is detected, 0 otherwise", and its204 labels include `partition_id`, `slice_id` ("ID of the associated slice"),205 and `state` ("Health of the partition (e.g., HEALTHY, UNHEALTHY)"). Run the206 query for each `{partition_id}` recorded on `{slice_name}`'s nodes in Step 2:207208```promql209kubernetes_io:accelerator_partition_state{210 monitored_resource="k8s_node_pool",211 cluster_name="{cluster_name}",212 partition_id="{partition_id}"213}214```215216 - Read only time series whose value is `1` (where `state="HEALTHY"` means the217 partition is healthy and functional, per218 [Verify the status of the nodes and the partitions](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)).219 - Because220 [Kubernetes metrics](https://docs.cloud.google.com/monitoring/api/metrics_kubernetes.md.txt)221 defines `slice_id` only as "ID of the associated slice", always confirm the222 conflicting slice from the `Slice` custom resources in the cluster rather223 than from `slice_id` alone: list the `Slice` custom resources as shown in224 [Delete the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),225 describe each one following226 [Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),227 and check which slice's `Spec` > `Partition Ids` includes the partition ID228 (or any other partition ID of any shape on the same nodes from Step 2).2293. **Next steps based on Step 3 results**:230 - **Another slice uses the nodes now**: Proceed to Step 4.231 - **No other slice uses the nodes now, but the metric (value `1`) showed an232 associated `slice_id` around `{issue_time}`**: The conflict may already be233 over. Proceed to Step 4, item 1 (to re-create or retry `{slice_name}`) and234 then Step 4, item 4.235 - **Neither the current `Slice` custom resources nor the metric around236 `{issue_time}` shows another slice**: Node overlap isn't confirmed. Hand237 off to `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring` or escalate238 to Cloud Customer Care (Step 4, item 4).239240---241242### Step 4: Resolution `[High Risk]`243244> **Guardrails**:245> - **Never** delete or change the other slice, or stop its workload, to free246> the nodes. Deleting a `Slice` custom resource starts dismantling that slice247> (`DEACTIVATING`, see248> [Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)).249> The owner of that slice decides when to release it.250> - **Never** manually modify or delete `Slice` custom resources created by251> Kueue. Per252> [Target your node pool to belong to a block or a sub-block](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),253> "Avoid manually modifying Slice custom resources created by Kueue."254> - **Never** change GKE-managed instance groups or VMs directly through Compute255> Engine, and never delete node pools to free a partition. Manage node pools256> and slices through GKE.257258First, check whether Kueue manages `{slice_name}` or the conflicting slice:259260- Per261 [Monitor the status of the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt),262 when a slice is created or actively used by a workload, the slice controller263 attaches the `kueue.gke.io/tpu-slice-used-by-namespace` and264 `kueue.gke.io/tpu-slice-used-by-workload` labels.265- Per266 [Optimize workload scheduling with slice reuse](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)267 and268 [Enable Slice reuse](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt),269 when slice reuse is enabled, "Kueue marks the slice as idle at the end of a270 job instead of deleting it", and "Reusable slices are named dynamically with271 an `s-<topology>` prefix, decoupling them from specific workload names."272273Then apply the matching option:2742751. **Wait for the conflicting slice to release the nodes and retry**:276 - **If Kueue manages the conflicting slice** (it has the `kueue.gke.io/...`277 labels or an `s-<topology>` name prefix): Do not suggest deleting the278 `Slice` custom resource directly. Point the workload owner to279 [Clean up](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)280 (deleting the `JobSet` triggers Kueue to delete the associated `Slice`281 custom resources) and, if slice reuse keeps an idle `s-<topology>` slice282 allocated after a job ends, to283 [Enforce slice deletion per workload](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)284 (`kueue.gke.io/tpu-enforce-slice-deletion: "true"`).285 - **If a custom scheduler or user created the conflicting slice directly**:286 Its owner can delete it when finished, as described in287 [Delete the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).288 - **How `{slice_name}` retries**: Check whether `{slice_name}` already has289 `slice.gke.io/retry-on-failure: "true"` in `Metadata` > `Annotations`. Per290 [Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),291 when a `Slice` custom resource is created with this annotation, "If the292 slice isn't created because of the `SliceCreationFailed` status reason, the293 controller will retry until the slice is successfully formed." Because the294 documentation only shows configuring295 `slice.gke.io/retry-on-failure: "true"` in the `Slice` manifest when296 creating a `Slice` custom resource (and doesn't state that adding the297 annotation to an existing slice already in `SliceCreationFailed` starts a298 retry), if `{slice_name}` wasn't created by Kueue and lacks the annotation,299 tell the user to delete `{slice_name}` following300 [Delete the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)301 and re-create it with `slice.gke.io/retry-on-failure: "true"` in its302 manifest following303 [Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).3042. **Use available partitions whose nodes no other slice uses**: If the user305 manages `Slice` custom resources directly (not through Kueue) and can't wait,306 direct them to307 [Verify the status of the nodes and the partitions](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)308 and309 [Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)310 so they can pick available partitions (`HEALTHY` partitions, or `DEGRADED`311 `4x4x4` partitions) whose nodes aren't used by any other slice (across all312 partition shapes on those nodes) and create the `Slice` custom resource with313 them.3143. **Schedule dynamic slices with Kueue and TAS instead of creating `Slice`315 custom resources directly**: Per316 [Target your node pool to belong to a block or a sub-block](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)317 and318 [Create a dynamic slice with Kueue and TAS](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt),319 instead of creating `Slice` custom resources directly, teams can schedule320 GKE workloads with Kueue and Topology Aware Scheduling (TAS), where "Kueue321 automatically handles the creation and deletion of Slice custom resources"322 and admits a workload after its slice becomes active.3234. **Verify or escalate**: Repeat Step 1 following324 [Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt):325 - **`ACTIVE`**: The slice is fully formed, healthy, and ready to execute326 workloads.327 - **`ACTIVE_DEGRADED`**: The slice is formed, and workloads can run, but328 performance might be impacted.329 - **Still `SliceCreationFailed` after no other slice uses the nodes, or330 `FAILED`**: Hand off to331 `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`, or advise the user332 to contact Cloud Customer Care following333 [Before you contact support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)334 and335 [Recommendations for efficient support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt).