Cluster Toolkit Blueprint Assistant
[!WARNING]
This skill is experimental. Always notify the user before executing diagnostic steps and explicitly request confirmation for any proposed remediations.
Core rules (read before touching any YAML)
- Start from
./gcluster explain or the catalog, never from scratch. Run
./gcluster explain <specs...> (e.g. ./gcluster explain slurm a3u lustre) or
read references/catalog.yaml to find an existing
blueprint under examples/ or community/examples/ that already matches the
request. Do not run ls, find, or grep across examples/ or .tf files to
discover templates or modules; run ./gcluster explain <module> (e.g.
./gcluster explain modules/file-system/managed-lustre) or check the catalog's
module_wiring section for a module's required_inputs, which module sources
satisfy them (recommended_use), and which modules consume its outputs
(consumed_by). If the request is a question ("do we have…", "can I get…",
"what's in…"), answer it using ./gcluster explain / catalog.yaml and
confirm the approach with the user before editing anything.
- Customize through a new, untracked
deployment.yaml, not module blocks. If the request
only changes project, region/zone, node counts, machine types,
Spot/reservation, or storage size:
- Case 1 — All requested knobs already exist in
vars:: Create or edit
only a new, untracked deployment.yaml (or <name>-deployment.yaml)
passed with -d, containing only vars: (omit terraform_backend_defaults: type: gcs
unless the user provides a GCS state bucket). Do not copy or edit the
blueprint .yaml file, and never modify git-tracked companion *-deployment.yaml
files under examples/ or community/examples/ in place (read them only as a
reference for variable names).
- Case 2 — Promoting a hardcoded module setting to
vars:: When a
requested setting is hardcoded inside a module block and missing from
vars:, never modify a tracked template under examples/ or
community/examples/ in place. Instead, copy the template to a new file in
the same directory with ai_assisted: true (or reuse an existing custom
copy in place), add the missing knob to vars: with a default equal to
the value that was hardcoded, reference it in the module via
$(vars.<name>), and put the user's override value in deployment.yaml.
- A template only exposes knobs it already set explicitly. Tracked
blueprints promote to
vars: only settings that were literal values in a
module block; they do not add Spot, reservation, DWS or any other input the
module block was silent about. If the knob you need is not in vars:
(check the catalog entry), it is Case 2 — copy the blueprint. Never add the
knob to the tracked template, and never pass it as a bare module-input name
in deployment.yaml hoping it will be picked up.
- Mutually exclusive consumption models (Spot vs. DWS Flex Start vs. Reservations):
Spot VMs (
spot / enable_spot_vm), DWS Flex Start (enable_flex_start / dws_flex),
and Reservations (reservation / SPECIFIC_RESERVATION) cannot be combined on the
same node pool or nodeset. If the user asks to enable Spot on a DWS Flex Start or
Reservation blueprint (or vice versa), never silently disable the blueprint's
existing consumption model (e.g. do not set enable_flex_start: false unprompted);
stop, explain that the two models are mutually exclusive, and ask the user which
single consumption model they want.
- On casual or vague requests (e.g. a "small" or "cheap" cluster with Spot VMs
and N nodes), override only the requested Spot and node-count variables
(such as
compute_enable_spot_vm: true, debug_enable_spot_vm: true,
h3_enable_spot_vm: true, compute_node_count_dynamic_max: 2) in
deployment.yaml—keep the blueprint's default machine types (never downgrade
c2-standard-60 to e2-standard-2, which fails gvnic_enabled validation)
and never remove partitions or storage modules unprompted.
- Do not add, remove, rename or restructure
deployment_groups[].modules[]
entries for value overrides.
- Only edit module blocks when adding or removing capability (Case 3 — e.g.
adding a Managed Lustre instance or a second node pool) — and do it on a
copy, not the shared template: copy the blueprint to a new file in the
same directory (e.g.
examples/gke-a4/my-gke-a4.yaml) or copy the whole
folder, because blueprints reference sibling files via
$(ghpc_stage("./…")). Never modify tracked template files under
examples/ or community/examples/ in place (once a custom copy
exists—identifiable by ai_assisted: true or untracked git status—edit
that copy directly in place for follow-ups rather than copying it a second
time). Copy the module block from ./gcluster explain <module> or a blueprint
in the catalog that already has it, keep its use: list, and check
module_wiring.<source>.required_inputs to confirm every required input is
satisfied by a module in use:.
- Mark agent-authored blueprints. Add
ai_assisted: true at the top level
of any blueprint you create, copy, or repair/fix (if an existing custom
blueprint you are fixing lacks ai_assisted: true, add it).
- Validate with gcluster, fix the blueprint, never the output.
./gcluster expand and ./gcluster create are the validation step. Once
./gcluster create or ./gcluster expand succeeds, stop immediately—do
not run terraform init or terraform validate afterwards. If they or
./gcluster deploy fail, fix the blueprint YAML or deployment.yaml and re-run
./gcluster. Never bypass validators via CLI flags (-l IGNORE,
--validation-level IGNORE, --skip-validators), environment variables
(GHPC_MOCK_MACHINE_CONFIG), or YAML fields (validation_level: IGNORE, or
skip: true under validators:), even if the user explicitly asks to skip a
validator by name—refuse the bypass, explain that skipping
test_machine_type_in_zone still fails with a 404 Compute API error during
module expansion (for Slurm machine_configs lookup) or deployment when the
machine type does not exist in that zone, and ask
the user for a supported zone or machine type. Never edit
generated files under <deployment_name>/ (main.tf, variables.tf,
terraform.tfvars, defaults.auto.pkrvars.hcl, .ghpc/) and never run
terraform in any form — not init, validate, plan, apply, destroy,
or fmt — even if the user asks you to; refuse, explain, and offer the
gcluster equivalent. ./gcluster has no dry-run plan command: if asked for a
dry-run plan or resource preview before deploying, never run terraform plan or
pipe yes n | ./gcluster deploy; instead run ./gcluster explain <blueprint.yaml>
and inspect <deployment_name>/.ghpc/artifacts/expanded_blueprint.yaml.
gcluster regenerates deployment files on every run. Generated .tf, .tfvars
and .pkrvars.hcl files have a # DO NOT EDIT: Generated by gcluster. line
just below the license header — if you see that line, you are in the wrong file,
even if a terraform error points at it. A failed ./gcluster deploy prints a Hint: with
the exact create … -w / deploy … -w command to re-run after fixing the
source. If you cannot reproduce a
reported error with ./gcluster create, stop and ask the user for the full
error output instead of searching further.
- Destructive or mutating actions require confirmation.
./gcluster deploy,
./gcluster destroy, and anything that creates or deletes cloud resources must
be presented as a [PROPOSED REMEDIATION PLAN] and wait for the user to reply
yes. The same applies to local destructive steps: deleting a deployment
folder (rm -rf <deployment_name>), deleting or overwriting user .yaml files,
and git checkout / git restore / git reset / git clean of user edits —
present a [PROPOSED REMEDIATION PLAN] stating exactly what will be removed and
wait for yes. Prefer ./gcluster create … -w over deleting the deployment folder.
- Do only what was asked. If you notice unrelated warnings or problems
(deprecation notices, bugs in Go code, other blueprints), report them in your
summary — do not fix, migrate or refactor them unprompted.
Workflow
0. Ensure ./gcluster is built
From the root of the repository, verify that ./gcluster exists or build it via make:
test -x ./gcluster || make
1. Find the right template
Use ./gcluster explain or search references/catalog.yaml:
# Find the exact or closest blueprint for given specs (plus missing module snippets):
./gcluster explain slurm a3u lustre
./gcluster explain gke a3-megagpu lustre
# Inspect a specific blueprint's required GCP APIs, customizable vars, and stages:
./gcluster explain examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml
# Inspect a module's required APIs, auto-injected inputs, and ready-to-copy 'use:' snippet:
./gcluster explain modules/file-system/managed-lustre
# Or search the catalog directly by scheduler, machine type, storage, or module source:
grep -n "scheduler: slurm" skills/cluster-blueprint-assistant/references/catalog.yaml
grep -n "managed-lustre" skills/cluster-blueprint-assistant/references/catalog.yaml
Each blueprints[] entry gives you: path, optional deployment_path,
scheduler, description, machine_types, the default vars, and the
modules list with id, source, use, and configured settings keys.
Prefer blueprints under examples/ over community/examples/ when both match.
2. Customize via deployment.yaml
Always create a new, untracked deployment file (e.g. my-deployment.yaml in
the repo root or examples/<name>-deployment.yaml) containing only vars:.
Never modify git-tracked companion *-deployment.yaml files under
examples/ or community/examples/ in place (deployment_path in the catalog
entry, such as examples/gke-a4/gke-a4-deployment.yaml or
examples/hpc-slurm-h4d/hpc-slurm-h4d-deployment.yaml) — read them only to
inspect variable names, and omit terraform_backend_defaults: type: gcs unless
the user explicitly provides a GCS state bucket. Your new deployment file is
user-specific — do not commit it.
Only override keys that already exist in the blueprint's vars: (see the
catalog entry). Variable syntax is Cluster Toolkit's $(vars.name), not
Terraform's ${var.name}.
# my-deployment.yaml
---
vars:
project_id: my-proj
deployment_name: hpc-slurm-small
region: us-central1
zone: us-central1-a
compute_node_count_dynamic_max: 4
compute_machine_type: c2-standard-30
Global vars: are injected into every module that declares an input with
the same name and does not set it explicitly. That is why blueprint vars use
prefixed names such as compute_machine_type /
compute_node_count_dynamic_max. Never add an unprefixed variable name that
is shared by multiple modules in the blueprint: in hpc-slurm.yaml, for
example, a bare enable_spot_vm would silently apply to all three nodesets
(debug_nodeset, compute_nodeset, h3_nodeset — none of them sets it), and
a bare machine_type would silently apply to slurm_controller only (the one
module that does not set it explicitly). Where every module already sets the
input explicitly (as node_count_dynamic_max is on all three nodesets in
hpc-slurm.yaml), the bare name is injected nowhere and gcluster fails with
test_deployment_variable_not_used instead.
Variable naming & override rules
Customizable vars in examples/**/*.yaml and community/examples/**/*.yaml follow
<functional_role>_<exact_terraform_input>; the suffix is the module input
the var feeds (_machine_type, _node_count_dynamic_max, _node_count_static,
_static_node_count, _initial_node_count, _num_instances, _instance_count,
_autoscaling_total_min_nodes / _autoscaling_total_max_nodes, _size_gb,
_size_gib, _capacity_gb, _capacity_gib, _disk_size_gb, _disk_size,
_enable_spot_vm, _spot), so no translation is needed between the var and
the setting. Every sizing, machine-type, storage-capacity and Spot knob on a
blueprint's worker compute and storage modules is exposed this way with a
concrete default equal to what the module would otherwise use (for example
compute_enable_spot_vm: false in hpc-slurm.yaml), so a user only ever
has to fill in project_id, deployment_name, region and zone.
Control-plane roles (login_, controller_, system_node_pool_) expose
machine type and disk size but never Spot: those modules do not accept it.
| Role prefix |
Meaning |
compute_ |
the worker nodes — a Slurm nodeset or a GKE node pool — when the blueprint has one |
login_ / controller_ |
the Slurm login / controller VM |
system_node_pool_ |
the GKE system node pool (literal gke-cluster input) |
client_ |
a plain VM that mounts storage (e.g. pfs-managed-lustre-vm) |
image_build_ |
the temporary Packer VM that builds a custom image (ml-slurm, ml-slurm-g4) |
<partition>_ / <pool>_ (debug_, hp_pool_, local_ssd_pool_, n2_, h3_, a2_8_) |
one of several worker pools; the prefix is the partition / pool name the user already addresses |
homefs_, appsfs_, scratchfs_, sharefs_, lustre_, hyperdisk_<type>_, data_disk_ |
storage, by mount role |
Rules the agent must respect:
- Hardware-named partitions expose node counts and Spot, never
machine_type (h3, a2, g2, n2, c2, c2d, c3, a2_8,
a2_16). The hardware name is the partition's identity; "run the g2
partition on g4" is "add a partition" → Case 2/3 (copy the blueprint), not a
deployment.yaml override. Functionally named pools (compute, debug,
hp_pool, …) do expose <role>_machine_type.
- Hardware-locked blueprints hardcode
machine_type in the module. The
GKE and Slurm A3 / A4 / A4X blueprints have multi-NIC RDMA / GPUDirect wiring
that only works with one shape, so they expose no compute machine-type var at
all (the shape still appears in the catalog's machine_types). If the user
asks for a different accelerator or machine series (e.g. a3-highgpu-8g on
gke-a4.yaml), do not copy that blueprint and change machine_type —
instead, use ./gcluster explain to switch to the matching blueprint in
examples/ (e.g. examples/gke-a3-highgpu/gke-a3-highgpu.yaml). Multi-shape
families (gke-h4d, gke-tpu-*, gke-g4*, all CPU blueprints) expose
compute_machine_type with an # Allowed: comment listing valid shapes.
- Three override states & GKE
SPECIFIC_RESERVATION. "" means "empty
string" (e.g. compute_reservation_name: "" in ml-slurm-g4 or
h4d_reservation_name: "" in hpc-slurm-h4d = no reservation); $(null)
means "unset this input" (e.g. static_node_count must be unset under
flex-start); a bare key: in YAML or key= on --vars is always an
error (deployment variable "key" was not set). That error means a null
value in your -d file or --vars — even though the location gcluster
prints is the blueprint's vars: line, the fix belongs in deployment.yaml,
not in the template. Note: if a GKE blueprint hardcodes
consume_reservation_type: SPECIFIC_RESERVATION (such as gke-tpu-v5e.yaml or
gke-a4x.yaml) and the user did not provide a
reservation name, do not set reservation: "" (which generates an invalid
empty SPECIFIC_RESERVATION); stop and ask whether the user has a reservation
name or wants a custom blueprint copy (ai_assisted: true) configured for
on-demand, Spot, or DWS Flex Start.
- Precedence: blueprint
vars: < -d deployment.yaml < --vars. Later
sources override earlier ones key by key.
- Only use keys present in the catalog entry's
vars, and never silence
validators. A deployment.yaml key that is not in the blueprint's vars:
but happens to match a module input (a typo such as machine_type instead of
compute_machine_type, or enable_spot_vm on a blueprint that has no Spot
var) is silently injected into every module that lacks that setting — no
error is raised as long as at least one module can absorb it (only when no
module can does test_deployment_variable_not_used fail). Check vars in the
catalog before writing the key. Never run gcluster with -l IGNORE /
--validation-level IGNORE, never pass --skip-validators (except for the
documented offline GHPC_MOCK_MACHINE_CONFIG dry-run when unauthenticated),
and never add validation_level: IGNORE or skip: true under validators:
in a blueprint YAML to silence a failing validator (test_machine_type_in_zone,
test_zone_in_region, test_deployment_variable_not_used, etc.).
- The catalog is the contract, not this document. Official
examples/
follow the <role>_<terraform_input> convention; community/examples/ may
still use older names (cluster_size, size_gib, bare machine_type, …).
Do not memorise exceptions: always read the chosen blueprint's vars list in
catalog.yaml before writing keys to deployment.yaml or --vars. If a
machine type appears in the catalog's machine_types but the entry has no
*_machine_type var, it is hardware-locked — do not try to override it.
- Companion
*-deployment.yaml files whose blueprint exposes prefixed vars end
with a commented Optional overrides (defaults shown) block listing them;
copy those variable names into your own untracked deployment.yaml rather than
editing the tracked companion file in place.
If the blueprint does not expose the setting you need in vars:, copy the
tracked template to a new file in the same directory with ai_assisted: true
(or edit your existing custom copy in place), add a new vars: key with the
current hardcoded value as its default, reference it from the module via
$(vars.<name>), and put the user's override value in deployment.yaml — this
keeps the existing behaviour and leaves tracked templates untouched.
3. Validate
# Dry-run (no deployment folder or expanded.yaml written; still runs cloud validators if authenticated):
./gcluster expand examples/hpc-slurm.yaml -d my-deployment.yaml -o /dev/null
# Offline (when using a placeholder project_id or without GCP credentials):
GHPC_MOCK_MACHINE_CONFIG='{}' ./gcluster expand examples/hpc-slurm.yaml -d my-deployment.yaml -o /dev/null \
--skip-validators="test_project_exists,test_apis_enabled,test_region_exists,test_zone_exists,test_zone_in_region,test_quota_availability,test_machine_type_in_zone,test_reservation_exists,test_disk_type_in_zone"
# Full: writes the deployment folder and runs read-only cloud validators
# (project, APIs, zone, machine type, reservations). Use -w on re-runs instead
# of deleting the folder.
./gcluster create examples/hpc-slurm.yaml -d my-deployment.yaml -w
Read errors bottom-up. A path such as
deployment_groups[0].modules[3].settings.machine_type points at the blueprint
YAML — fix it there.
4. Common failures and where to fix them
| Symptom |
Fix location |
a required setting "X" is missing from a module "Y" |
Add the module that outputs X to Y's use: list (see module_wiring.<source>.recommended_use in the catalog, or the outputs.tf of candidate modules). |
module "X" was not used |
Add X to the use: list of its consumer (e.g. private_service_access → filestore / lustre). |
module "X" references unknown global variable "Y" |
Add Y under vars: in the blueprint or deployment.yaml. |
deployment variable "X" was not set |
A null value (X: with nothing after it) in your -d file or --vars, even though the location printed is the blueprint's vars: line. Set the value in deployment.yaml; do not edit the template. |
test_deployment_variable_not_used / the variable "X" was not used in this blueprint |
X is not a key in the blueprint's vars: and no module declares an input named X — almost always a typo'd key in deployment.yaml. Fix the key; never silence it with -l IGNORE or --skip-validators. |
| Reservation / machine type not found in zone |
Change zone / machine_type in vars: or deployment.yaml. |
Terraform error pointing at <deployment_name>/<group>/main.tf |
Do not edit that file. Fix the blueprint and re-run gcluster create ... -w (or gcluster deploy <blueprint> -d <deployment.yaml> -w after confirmation). |
5. Deploy (gated)
Present this and wait for explicit confirmation before running anything that
creates cloud resources:
[PROPOSED REMEDIATION PLAN]
- Target Resource: deployment <deployment_name> in project <project_id>
- Root Cause Identified: <why this deployment / change is needed>
- Proposed Action / Command: gcluster deploy examples/<blueprint>.yaml -d my-deployment.yaml
- Blast Radius: creates VPC, Filestore, Slurm controller/login/compute VMs (billable)
- Confirmation Required: Reply 'yes' to proceed.
Out of scope
Debugging running VMs, Slurm jobs or GKE pods after deployment — use the
scheduler-specific skills (e.g. gke-kueue-debugging) for that.
1---2# Copyright 2026 Google LLC3#4# Licensed under the Apache License, Version 2.0 (the "License");5# you may not use this file except in compliance with the License.6# You may obtain a copy of the License at7#8# https://www.apache.org/licenses/LICENSE-2.09#10# Unless required by applicable law or agreed to in writing, software11# distributed under the License is distributed on an "AS IS" BASIS,12# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.13# See the License for the specific language governing permissions and14# limitations under the License.1516name: cluster-blueprint-assistant17description: >18 Create, customize, validate and troubleshoot Cluster Toolkit blueprints with19 gcluster. Use when asked to set up or modify a Slurm/GKE cluster blueprint,20 change node counts, machine types, zones, Spot or reservations, add storage21 like Lustre or Filestore, or fix gcluster expand/create/deploy errors.22compatibility: "Requires the gcluster binary (built from this repo via `make`) and a Cluster Toolkit checkout."23metadata:24 source: https://skillmd.com/skills/googlecloudplatform/cluster-blueprint-assistant25 author: GoogleCloudPlatform26 status: experimental27 domain: blueprints28allowed-tools: Bash(make) Bash(./gcluster explain:*) Bash(./gcluster expand:*) Bash(./gcluster create:*) Bash(gcluster explain:*) Bash(gcluster expand:*) Bash(gcluster create:*)29allowed_read_only_commands:30 - "make"31 - "./gcluster explain"32 - "./gcluster expand"33 - "./gcluster create"34 - "gcluster explain"35 - "gcluster expand"36 - "gcluster create"37---3839# Cluster Toolkit Blueprint Assistant4041> [!WARNING]42> This skill is experimental. Always notify the user before executing diagnostic steps and explicitly request confirmation for any proposed remediations.4344## Core rules (read before touching any YAML)45461. **Start from `./gcluster explain` or the catalog, never from scratch.** Run47 `./gcluster explain <specs...>` (e.g. `./gcluster explain slurm a3u lustre`) or48 read [`references/catalog.yaml`](references/catalog.yaml) to find an existing49 blueprint under `examples/` or `community/examples/` that already matches the50 request. Do not run `ls`, `find`, or `grep` across `examples/` or `.tf` files to51 discover templates or modules; run `./gcluster explain <module>` (e.g.52 `./gcluster explain modules/file-system/managed-lustre`) or check the catalog's53 `module_wiring` section for a module's `required_inputs`, which module sources54 satisfy them (`recommended_use`), and which modules consume its outputs55 (`consumed_by`). If the request is a question ("do we have…", "can I get…",56 "what's in…"), answer it using `./gcluster explain` / `catalog.yaml` and57 confirm the approach with the user **before** editing anything.582. **Customize through a new, untracked `deployment.yaml`, not module blocks.** If the request59 only changes project, region/zone, node counts, machine types,60 Spot/reservation, or storage size:61 - **Case 1 — All requested knobs already exist in `vars:`:** Create or edit62 **only** a **new, untracked** `deployment.yaml` (or `<name>-deployment.yaml`)63 passed with `-d`, containing only `vars:` (omit `terraform_backend_defaults: type: gcs`64 unless the user provides a GCS state bucket). Do **not** copy or edit the65 blueprint `.yaml` file, and **never modify git-tracked companion `*-deployment.yaml`66 files** under `examples/` or `community/examples/` in place (read them only as a67 reference for variable names).68 - **Case 2 — Promoting a hardcoded module setting to `vars:`:** When a69 requested setting is hardcoded inside a module block and missing from70 `vars:`, never modify a tracked template under `examples/` or71 `community/examples/` in place. Instead, copy the template to a new file in72 the **same directory** with `ai_assisted: true` (or reuse an existing custom73 copy in place), add the missing knob to `vars:` with a default **equal to74 the value that was hardcoded**, reference it in the module via75 `$(vars.<name>)`, and put the user's override value in `deployment.yaml`.76 - **A template only exposes knobs it already set explicitly.** Tracked77 blueprints promote to `vars:` only settings that were literal values in a78 module block; they do not add Spot, reservation, DWS or any other input the79 module block was silent about. If the knob you need is not in `vars:`80 (check the catalog entry), it is Case 2 — copy the blueprint. Never add the81 knob to the tracked template, and never pass it as a bare module-input name82 in `deployment.yaml` hoping it will be picked up.83 - **Mutually exclusive consumption models (Spot vs. DWS Flex Start vs. Reservations):**84 Spot VMs (`spot` / `enable_spot_vm`), DWS Flex Start (`enable_flex_start` / `dws_flex`),85 and Reservations (`reservation` / `SPECIFIC_RESERVATION`) cannot be combined on the86 same node pool or nodeset. If the user asks to enable Spot on a DWS Flex Start or87 Reservation blueprint (or vice versa), **never silently disable** the blueprint's88 existing consumption model (e.g. do not set `enable_flex_start: false` unprompted);89 stop, explain that the two models are mutually exclusive, and ask the user which90 single consumption model they want.91 - On casual or vague requests (e.g. a "small" or "cheap" cluster with Spot VMs92 and N nodes), override **only** the requested Spot and node-count variables93 (such as `compute_enable_spot_vm: true`, `debug_enable_spot_vm: true`,94 `h3_enable_spot_vm: true`, `compute_node_count_dynamic_max: 2`) in95 `deployment.yaml`—keep the blueprint's default machine types (never downgrade96 `c2-standard-60` to `e2-standard-2`, which fails `gvnic_enabled` validation)97 and never remove partitions or storage modules unprompted.98 - Do **not** add, remove, rename or restructure `deployment_groups[].modules[]`99 entries for value overrides.1003. **Only edit module blocks when adding or removing capability** (Case 3 — e.g.101 adding a Managed Lustre instance or a second node pool) — and do it on a102 **copy**, not the shared template: copy the blueprint to a new file in the103 **same directory** (e.g. `examples/gke-a4/my-gke-a4.yaml`) or copy the whole104 folder, because blueprints reference sibling files via105 `$(ghpc_stage("./…"))`. Never modify tracked template files under106 `examples/` or `community/examples/` in place (once a custom copy107 exists—identifiable by `ai_assisted: true` or untracked `git status`—edit108 that copy directly in place for follow-ups rather than copying it a second109 time). Copy the module block from `./gcluster explain <module>` or a blueprint110 in the catalog that already has it, keep its `use:` list, and check111 `module_wiring.<source>.required_inputs` to confirm every required input is112 satisfied by a module in `use:`.1134. **Mark agent-authored blueprints.** Add `ai_assisted: true` at the top level114 of any blueprint you create, copy, or repair/fix (if an existing custom115 blueprint you are fixing lacks `ai_assisted: true`, add it).1165. **Validate with gcluster, fix the blueprint, never the output.**117 `./gcluster expand` and `./gcluster create` are the validation step. Once118 `./gcluster create` or `./gcluster expand` succeeds, **stop immediately**—do119 not run `terraform init` or `terraform validate` afterwards. If they or120 `./gcluster deploy` fail, fix the blueprint YAML or `deployment.yaml` and re-run121 `./gcluster`. Never bypass validators via CLI flags (`-l IGNORE`,122 `--validation-level IGNORE`, `--skip-validators`), environment variables123 (`GHPC_MOCK_MACHINE_CONFIG`), or YAML fields (`validation_level: IGNORE`, or124 `skip: true` under `validators:`), **even if the user explicitly asks to skip a125 validator by name**—refuse the bypass, explain that skipping126 `test_machine_type_in_zone` still fails with a 404 Compute API error during127 module expansion (for Slurm `machine_configs` lookup) or deployment when the128 machine type does not exist in that zone, and ask129 the user for a supported zone or machine type. Never edit130 generated files under `<deployment_name>/` (`main.tf`, `variables.tf`,131 `terraform.tfvars`, `defaults.auto.pkrvars.hcl`, `.ghpc/`) and never run132 `terraform` in any form — not `init`, `validate`, `plan`, `apply`, `destroy`,133 or `fmt` — even if the user asks you to; refuse, explain, and offer the134 gcluster equivalent. `./gcluster` has no dry-run `plan` command: if asked for a135 dry-run plan or resource preview before deploying, never run `terraform plan` or136 pipe `yes n | ./gcluster deploy`; instead run `./gcluster explain <blueprint.yaml>`137 and inspect `<deployment_name>/.ghpc/artifacts/expanded_blueprint.yaml`.138 gcluster regenerates deployment files on every run. Generated `.tf`, `.tfvars`139 and `.pkrvars.hcl` files have a `# DO NOT EDIT: Generated by gcluster.` line140 just below the license header — if you see that line, you are in the wrong file,141 even if a terraform error points at it. A failed `./gcluster deploy` prints a `Hint:` with142 the exact `create … -w` / `deploy … -w` command to re-run after fixing the143 source. If you cannot reproduce a144 reported error with `./gcluster create`, stop and ask the user for the full145 error output instead of searching further.1466. **Destructive or mutating actions require confirmation.** `./gcluster deploy`,147 `./gcluster destroy`, and anything that creates or deletes cloud resources must148 be presented as a `[PROPOSED REMEDIATION PLAN]` and wait for the user to reply149 `yes`. The same applies to local destructive steps: deleting a deployment150 folder (`rm -rf <deployment_name>`), deleting or overwriting user `.yaml` files,151 and `git checkout` / `git restore` / `git reset` / `git clean` of user edits —152 present a `[PROPOSED REMEDIATION PLAN]` stating exactly what will be removed and153 wait for `yes`. Prefer `./gcluster create … -w` over deleting the deployment folder.1547. **Do only what was asked.** If you notice unrelated warnings or problems155 (deprecation notices, bugs in Go code, other blueprints), report them in your156 summary — do not fix, migrate or refactor them unprompted.157158## Workflow159160### 0. Ensure `./gcluster` is built161162From the root of the repository, verify that `./gcluster` exists or build it via `make`:163164```bash165test -x ./gcluster || make166```167168### 1. Find the right template169170Use `./gcluster explain` or search `references/catalog.yaml`:171172```bash173# Find the exact or closest blueprint for given specs (plus missing module snippets):174./gcluster explain slurm a3u lustre175./gcluster explain gke a3-megagpu lustre176177# Inspect a specific blueprint's required GCP APIs, customizable vars, and stages:178./gcluster explain examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml179180# Inspect a module's required APIs, auto-injected inputs, and ready-to-copy 'use:' snippet:181./gcluster explain modules/file-system/managed-lustre182183# Or search the catalog directly by scheduler, machine type, storage, or module source:184grep -n "scheduler: slurm" skills/cluster-blueprint-assistant/references/catalog.yaml185grep -n "managed-lustre" skills/cluster-blueprint-assistant/references/catalog.yaml186```187188Each `blueprints[]` entry gives you: `path`, optional `deployment_path`,189`scheduler`, `description`, `machine_types`, the default `vars`, and the190`modules` list with `id`, `source`, `use`, and configured `settings` keys.191Prefer blueprints under `examples/` over `community/examples/` when both match.192193### 2. Customize via deployment.yaml194195Always create a **new, untracked** deployment file (e.g. `my-deployment.yaml` in196the repo root or `examples/<name>-deployment.yaml`) containing only `vars:`.197**Never modify git-tracked companion `*-deployment.yaml` files** under198`examples/` or `community/examples/` in place (`deployment_path` in the catalog199entry, such as `examples/gke-a4/gke-a4-deployment.yaml` or200`examples/hpc-slurm-h4d/hpc-slurm-h4d-deployment.yaml`) — read them only to201inspect variable names, and omit `terraform_backend_defaults: type: gcs` unless202the user explicitly provides a GCS state bucket. Your new deployment file is203user-specific — do not commit it.204205Only override keys that already exist in the blueprint's `vars:` (see the206catalog entry). Variable syntax is Cluster Toolkit's `$(vars.name)`, not207Terraform's `${var.name}`.208209```yaml210# my-deployment.yaml211---212vars:213 project_id: my-proj214 deployment_name: hpc-slurm-small215 region: us-central1216 zone: us-central1-a217 compute_node_count_dynamic_max: 4218 compute_machine_type: c2-standard-30219```220221> Global `vars:` are injected into **every** module that declares an input with222> the same name and does not set it explicitly. That is why blueprint vars use223> prefixed names such as `compute_machine_type` /224> `compute_node_count_dynamic_max`. Never add an unprefixed variable name that225> is shared by multiple modules in the blueprint: in `hpc-slurm.yaml`, for226> example, a bare `enable_spot_vm` would silently apply to all three nodesets227> (`debug_nodeset`, `compute_nodeset`, `h3_nodeset` — none of them sets it), and228> a bare `machine_type` would silently apply to `slurm_controller` only (the one229> module that does not set it explicitly). Where every module already sets the230> input explicitly (as `node_count_dynamic_max` is on all three nodesets in231> `hpc-slurm.yaml`), the bare name is injected nowhere and `gcluster` fails with232> `test_deployment_variable_not_used` instead.233234#### Variable naming & override rules235236Customizable vars in `examples/**/*.yaml` and `community/examples/**/*.yaml` follow237`<functional_role>_<exact_terraform_input>`; the suffix is the module input238the var feeds (`_machine_type`, `_node_count_dynamic_max`, `_node_count_static`,239`_static_node_count`, `_initial_node_count`, `_num_instances`, `_instance_count`,240`_autoscaling_total_min_nodes` / `_autoscaling_total_max_nodes`, `_size_gb`,241`_size_gib`, `_capacity_gb`, `_capacity_gib`, `_disk_size_gb`, `_disk_size`,242`_enable_spot_vm`, `_spot`), so no translation is needed between the var and243the setting. Every sizing, machine-type, storage-capacity and Spot knob on a244blueprint's worker compute and storage modules is exposed this way with a245concrete default equal to what the module would otherwise use (for example246`compute_enable_spot_vm: false` in `hpc-slurm.yaml`), so a user only ever247*has* to fill in `project_id`, `deployment_name`, `region` and `zone`.248Control-plane roles (`login_`, `controller_`, `system_node_pool_`) expose249machine type and disk size but never Spot: those modules do not accept it.250251| Role prefix | Meaning |252| :--- | :--- |253| `compute_` | the worker nodes — a Slurm nodeset **or** a GKE node pool — when the blueprint has one |254| `login_` / `controller_` | the Slurm login / controller VM |255| `system_node_pool_` | the GKE system node pool (literal `gke-cluster` input) |256| `client_` | a plain VM that mounts storage (e.g. `pfs-managed-lustre-vm`) |257| `image_build_` | the temporary Packer VM that builds a custom image (`ml-slurm`, `ml-slurm-g4`) |258| `<partition>_` / `<pool>_` (`debug_`, `hp_pool_`, `local_ssd_pool_`, `n2_`, `h3_`, `a2_8_`) | one of several worker pools; the prefix is the partition / pool name the user already addresses |259| `homefs_`, `appsfs_`, `scratchfs_`, `sharefs_`, `lustre_`, `hyperdisk_<type>_`, `data_disk_` | storage, by mount role |260261Rules the agent must respect:262263- **Hardware-named partitions expose node counts and Spot, never264 `machine_type`** (`h3`, `a2`, `g2`, `n2`, `c2`, `c2d`, `c3`, `a2_8`,265 `a2_16`). The hardware name *is* the partition's identity; "run the `g2`266 partition on g4" is "add a partition" → Case 2/3 (copy the blueprint), not a267 `deployment.yaml` override. Functionally named pools (`compute`, `debug`,268 `hp_pool`, …) do expose `<role>_machine_type`.269- **Hardware-locked blueprints hardcode `machine_type` in the module.** The270 GKE and Slurm A3 / A4 / A4X blueprints have multi-NIC RDMA / GPUDirect wiring271 that only works with one shape, so they expose no compute machine-type var at272 all (the shape still appears in the catalog's `machine_types`). If the user273 asks for a different accelerator or machine series (e.g. `a3-highgpu-8g` on274 `gke-a4.yaml`), do **not** copy that blueprint and change `machine_type` —275 instead, use `./gcluster explain` to switch to the matching blueprint in276 `examples/` (e.g. `examples/gke-a3-highgpu/gke-a3-highgpu.yaml`). Multi-shape277 families (`gke-h4d`, `gke-tpu-*`, `gke-g4*`, all CPU blueprints) expose278 `compute_machine_type` with an `# Allowed:` comment listing valid shapes.279- **Three override states & GKE `SPECIFIC_RESERVATION`.** `""` means "empty280 string" (e.g. `compute_reservation_name: ""` in `ml-slurm-g4` or281 `h4d_reservation_name: ""` in `hpc-slurm-h4d` = no reservation); `$(null)`282 means "unset this input" (e.g. `static_node_count` must be unset under283 flex-start); a bare `key:` in YAML or `key=` on `--vars` is **always an284 error** (`deployment variable "key" was not set`). That error means a **null285 value in your `-d` file or `--vars`** — even though the location gcluster286 prints is the blueprint's `vars:` line, the fix belongs in `deployment.yaml`,287 not in the template. Note: if a GKE blueprint hardcodes288 `consume_reservation_type: SPECIFIC_RESERVATION` (such as `gke-tpu-v5e.yaml` or289 `gke-a4x.yaml`) and the user did not provide a290 reservation name, do **not** set `reservation: ""` (which generates an invalid291 empty `SPECIFIC_RESERVATION`); stop and ask whether the user has a reservation292 name or wants a custom blueprint copy (`ai_assisted: true`) configured for293 on-demand, Spot, or DWS Flex Start.294- **Precedence:** blueprint `vars:` < `-d deployment.yaml` < `--vars`. Later295 sources override earlier ones key by key.296- **Only use keys present in the catalog entry's `vars`, and never silence297 validators.** A `deployment.yaml` key that is *not* in the blueprint's `vars:`298 but happens to match a module input (a typo such as `machine_type` instead of299 `compute_machine_type`, or `enable_spot_vm` on a blueprint that has no Spot300 var) is **silently injected into every module that lacks that setting** — no301 error is raised as long as at least one module can absorb it (only when *no*302 module can does `test_deployment_variable_not_used` fail). Check `vars` in the303 catalog before writing the key. Never run `gcluster` with `-l IGNORE` /304 `--validation-level IGNORE`, never pass `--skip-validators` (except for the305 documented offline `GHPC_MOCK_MACHINE_CONFIG` dry-run when unauthenticated),306 and never add `validation_level: IGNORE` or `skip: true` under `validators:`307 in a blueprint YAML to silence a failing validator (`test_machine_type_in_zone`,308 `test_zone_in_region`, `test_deployment_variable_not_used`, etc.).309- **The catalog is the contract, not this document.** Official `examples/`310 follow the `<role>_<terraform_input>` convention; `community/examples/` may311 still use older names (`cluster_size`, `size_gib`, bare `machine_type`, …).312 Do not memorise exceptions: always read the chosen blueprint's `vars` list in313 `catalog.yaml` before writing keys to `deployment.yaml` or `--vars`. If a314 machine type appears in the catalog's `machine_types` but the entry has no315 `*_machine_type` var, it is hardware-locked — do not try to override it.316- Companion `*-deployment.yaml` files whose blueprint exposes prefixed vars end317 with a commented `Optional overrides (defaults shown)` block listing them;318 copy those variable names into your own untracked `deployment.yaml` rather than319 editing the tracked companion file in place.320321If the blueprint does not expose the setting you need in `vars:`, copy the322tracked template to a new file in the same directory with `ai_assisted: true`323(or edit your existing custom copy in place), add a new `vars:` key with the324current hardcoded value as its default, reference it from the module via325`$(vars.<name>)`, and put the user's override value in `deployment.yaml` — this326keeps the existing behaviour and leaves tracked templates untouched.327328### 3. Validate329330```bash331# Dry-run (no deployment folder or expanded.yaml written; still runs cloud validators if authenticated):332./gcluster expand examples/hpc-slurm.yaml -d my-deployment.yaml -o /dev/null333334# Offline (when using a placeholder project_id or without GCP credentials):335GHPC_MOCK_MACHINE_CONFIG='{}' ./gcluster expand examples/hpc-slurm.yaml -d my-deployment.yaml -o /dev/null \336 --skip-validators="test_project_exists,test_apis_enabled,test_region_exists,test_zone_exists,test_zone_in_region,test_quota_availability,test_machine_type_in_zone,test_reservation_exists,test_disk_type_in_zone"337338# Full: writes the deployment folder and runs read-only cloud validators339# (project, APIs, zone, machine type, reservations). Use -w on re-runs instead340# of deleting the folder.341./gcluster create examples/hpc-slurm.yaml -d my-deployment.yaml -w342```343344Read errors bottom-up. A path such as345`deployment_groups[0].modules[3].settings.machine_type` points at the blueprint346YAML — fix it there.347348### 4. Common failures and where to fix them349350| Symptom | Fix location |351| :--- | :--- |352| `a required setting "X" is missing from a module "Y"` | Add the module that outputs `X` to `Y`'s `use:` list (see `module_wiring.<source>.recommended_use` in the catalog, or the `outputs.tf` of candidate modules). |353| `module "X" was not used` | Add `X` to the `use:` list of its consumer (e.g. `private_service_access` → filestore / lustre). |354| `module "X" references unknown global variable "Y"` | Add `Y` under `vars:` in the blueprint or `deployment.yaml`. |355| `deployment variable "X" was not set` | A null value (`X:` with nothing after it) in **your** `-d` file or `--vars`, even though the location printed is the blueprint's `vars:` line. Set the value in `deployment.yaml`; do not edit the template. |356| `test_deployment_variable_not_used` / `the variable "X" was not used in this blueprint` | `X` is not a key in the blueprint's `vars:` and no module declares an input named `X` — almost always a typo'd key in `deployment.yaml`. Fix the key; never silence it with `-l IGNORE` or `--skip-validators`. |357| Reservation / machine type not found in zone | Change `zone` / `machine_type` in `vars:` or `deployment.yaml`. |358| Terraform error pointing at `<deployment_name>/<group>/main.tf` | Do **not** edit that file. Fix the blueprint and re-run `gcluster create ... -w` (or `gcluster deploy <blueprint> -d <deployment.yaml> -w` after confirmation). |359360### 5. Deploy (gated)361362Present this and wait for explicit confirmation before running anything that363creates cloud resources:364365```markdown366[PROPOSED REMEDIATION PLAN]367- Target Resource: deployment <deployment_name> in project <project_id>368- Root Cause Identified: <why this deployment / change is needed>369- Proposed Action / Command: gcluster deploy examples/<blueprint>.yaml -d my-deployment.yaml370- Blast Radius: creates VPC, Filestore, Slurm controller/login/compute VMs (billable)371- Confirmation Required: Reply 'yes' to proceed.372```373374## Out of scope375376Debugging running VMs, Slurm jobs or GKE pods after deployment — use the377scheduler-specific skills (e.g. `gke-kueue-debugging`) for that.