# Cluster Blueprint Assistant

> Create, customize, validate and troubleshoot Cluster Toolkit blueprints with gcluster. Use when asked to set up or modify a Slurm/GKE cluster blueprint, change node counts, machine types, zones, Spot or reservations, add storage like Lustre or Filestore, or fix gcluster expand/create/deploy errors.

- Skill: `googlecloudplatform/cluster-blueprint-assistant` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx get-skill add googlecloudplatform/cluster-blueprint-assistant`
- Raw SKILL.md: https://api.skillmd.com/api/skills/googlecloudplatform/cluster-blueprint-assistant/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: GoogleCloudPlatform (https://skillmd.com/u/googlecloudplatform)
- Updated: 2026-11-16
- Page: https://skillmd.com/skills/googlecloudplatform/cluster-blueprint-assistant

---


# Cluster Toolkit Blueprint Assistant

> [!WARNING]
> This skill is experimental. Always notify the user before executing diagnostic steps and explicitly request confirmation for any proposed remediations.

## Core rules (read before touching any YAML)

1. **Start from `./gcluster explain` or the catalog, never from scratch.** Run
   `./gcluster explain <specs...>` (e.g. `./gcluster explain slurm a3u lustre`) or
   read [`references/catalog.yaml`](references/catalog.yaml) to find an existing
   blueprint under `examples/` or `community/examples/` that already matches the
   request. Do not run `ls`, `find`, or `grep` across `examples/` or `.tf` files to
   discover templates or modules; run `./gcluster explain <module>` (e.g.
   `./gcluster explain modules/file-system/managed-lustre`) or check the catalog's
   `module_wiring` section for a module's `required_inputs`, which module sources
   satisfy them (`recommended_use`), and which modules consume its outputs
   (`consumed_by`). If the request is a question ("do we have…", "can I get…",
   "what's in…"), answer it using `./gcluster explain` / `catalog.yaml` and
   confirm the approach with the user **before** editing anything.
2. **Customize through a new, untracked `deployment.yaml`, not module blocks.** If the request
   only changes project, region/zone, node counts, machine types,
   Spot/reservation, or storage size:
   - **Case 1 — All requested knobs already exist in `vars:`:** Create or edit
     **only** a **new, untracked** `deployment.yaml` (or `<name>-deployment.yaml`)
     passed with `-d`, containing only `vars:` (omit `terraform_backend_defaults: type: gcs`
     unless the user provides a GCS state bucket). Do **not** copy or edit the
     blueprint `.yaml` file, and **never modify git-tracked companion `*-deployment.yaml`
     files** under `examples/` or `community/examples/` in place (read them only as a
     reference for variable names).
   - **Case 2 — Promoting a hardcoded module setting to `vars:`:** When a
     requested setting is hardcoded inside a module block and missing from
     `vars:`, never modify a tracked template under `examples/` or
     `community/examples/` in place. Instead, copy the template to a new file in
     the **same directory** with `ai_assisted: true` (or reuse an existing custom
     copy in place), add the missing knob to `vars:` with a default **equal to
     the value that was hardcoded**, reference it in the module via
     `$(vars.<name>)`, and put the user's override value in `deployment.yaml`.
   - **A template only exposes knobs it already set explicitly.** Tracked
     blueprints promote to `vars:` only settings that were literal values in a
     module block; they do not add Spot, reservation, DWS or any other input the
     module block was silent about. If the knob you need is not in `vars:`
     (check the catalog entry), it is Case 2 — copy the blueprint. Never add the
     knob to the tracked template, and never pass it as a bare module-input name
     in `deployment.yaml` hoping it will be picked up.
   - **Mutually exclusive consumption models (Spot vs. DWS Flex Start vs. Reservations):**
     Spot VMs (`spot` / `enable_spot_vm`), DWS Flex Start (`enable_flex_start` / `dws_flex`),
     and Reservations (`reservation` / `SPECIFIC_RESERVATION`) cannot be combined on the
     same node pool or nodeset. If the user asks to enable Spot on a DWS Flex Start or
     Reservation blueprint (or vice versa), **never silently disable** the blueprint's
     existing consumption model (e.g. do not set `enable_flex_start: false` unprompted);
     stop, explain that the two models are mutually exclusive, and ask the user which
     single consumption model they want.
   - On casual or vague requests (e.g. a "small" or "cheap" cluster with Spot VMs
     and N nodes), override **only** the requested Spot and node-count variables
     (such as `compute_enable_spot_vm: true`, `debug_enable_spot_vm: true`,
     `h3_enable_spot_vm: true`, `compute_node_count_dynamic_max: 2`) in
     `deployment.yaml`—keep the blueprint's default machine types (never downgrade
     `c2-standard-60` to `e2-standard-2`, which fails `gvnic_enabled` validation)
     and never remove partitions or storage modules unprompted.
   - Do **not** add, remove, rename or restructure `deployment_groups[].modules[]`
     entries for value overrides.
3. **Only edit module blocks when adding or removing capability** (Case 3 — e.g.
   adding a Managed Lustre instance or a second node pool) — and do it on a
   **copy**, not the shared template: copy the blueprint to a new file in the
   **same directory** (e.g. `examples/gke-a4/my-gke-a4.yaml`) or copy the whole
   folder, because blueprints reference sibling files via
   `$(ghpc_stage("./…"))`. Never modify tracked template files under
   `examples/` or `community/examples/` in place (once a custom copy
   exists—identifiable by `ai_assisted: true` or untracked `git status`—edit
   that copy directly in place for follow-ups rather than copying it a second
   time). Copy the module block from `./gcluster explain <module>` or a blueprint
   in the catalog that already has it, keep its `use:` list, and check
   `module_wiring.<source>.required_inputs` to confirm every required input is
   satisfied by a module in `use:`.
4. **Mark agent-authored blueprints.** Add `ai_assisted: true` at the top level
   of any blueprint you create, copy, or repair/fix (if an existing custom
   blueprint you are fixing lacks `ai_assisted: true`, add it).
5. **Validate with gcluster, fix the blueprint, never the output.**
   `./gcluster expand` and `./gcluster create` are the validation step. Once
   `./gcluster create` or `./gcluster expand` succeeds, **stop immediately**—do
   not run `terraform init` or `terraform validate` afterwards. If they or
   `./gcluster deploy` fail, fix the blueprint YAML or `deployment.yaml` and re-run
   `./gcluster`. Never bypass validators via CLI flags (`-l IGNORE`,
   `--validation-level IGNORE`, `--skip-validators`), environment variables
   (`GHPC_MOCK_MACHINE_CONFIG`), or YAML fields (`validation_level: IGNORE`, or
   `skip: true` under `validators:`), **even if the user explicitly asks to skip a
   validator by name**—refuse the bypass, explain that skipping
   `test_machine_type_in_zone` still fails with a 404 Compute API error during
   module expansion (for Slurm `machine_configs` lookup) or deployment when the
   machine type does not exist in that zone, and ask
   the user for a supported zone or machine type. Never edit
   generated files under `<deployment_name>/` (`main.tf`, `variables.tf`,
   `terraform.tfvars`, `defaults.auto.pkrvars.hcl`, `.ghpc/`) and never run
   `terraform` in any form — not `init`, `validate`, `plan`, `apply`, `destroy`,
   or `fmt` — even if the user asks you to; refuse, explain, and offer the
   gcluster equivalent. `./gcluster` has no dry-run `plan` command: if asked for a
   dry-run plan or resource preview before deploying, never run `terraform plan` or
   pipe `yes n | ./gcluster deploy`; instead run `./gcluster explain <blueprint.yaml>`
   and inspect `<deployment_name>/.ghpc/artifacts/expanded_blueprint.yaml`.
   gcluster regenerates deployment files on every run. Generated `.tf`, `.tfvars`
   and `.pkrvars.hcl` files have a `# DO NOT EDIT: Generated by gcluster.` line
   just below the license header — if you see that line, you are in the wrong file,
   even if a terraform error points at it. A failed `./gcluster deploy` prints a `Hint:` with
   the exact `create … -w` / `deploy … -w` command to re-run after fixing the
   source. If you cannot reproduce a
   reported error with `./gcluster create`, stop and ask the user for the full
   error output instead of searching further.
6. **Destructive or mutating actions require confirmation.** `./gcluster deploy`,
   `./gcluster destroy`, and anything that creates or deletes cloud resources must
   be presented as a `[PROPOSED REMEDIATION PLAN]` and wait for the user to reply
   `yes`. The same applies to local destructive steps: deleting a deployment
   folder (`rm -rf <deployment_name>`), deleting or overwriting user `.yaml` files,
   and `git checkout` / `git restore` / `git reset` / `git clean` of user edits —
   present a `[PROPOSED REMEDIATION PLAN]` stating exactly what will be removed and
   wait for `yes`. Prefer `./gcluster create … -w` over deleting the deployment folder.
7. **Do only what was asked.** If you notice unrelated warnings or problems
   (deprecation notices, bugs in Go code, other blueprints), report them in your
   summary — do not fix, migrate or refactor them unprompted.

## Workflow

### 0. Ensure `./gcluster` is built

From the root of the repository, verify that `./gcluster` exists or build it via `make`:

```bash
test -x ./gcluster || make
```

### 1. Find the right template

Use `./gcluster explain` or search `references/catalog.yaml`:

```bash
# Find the exact or closest blueprint for given specs (plus missing module snippets):
./gcluster explain slurm a3u lustre
./gcluster explain gke a3-megagpu lustre

# Inspect a specific blueprint's required GCP APIs, customizable vars, and stages:
./gcluster explain examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml

# Inspect a module's required APIs, auto-injected inputs, and ready-to-copy 'use:' snippet:
./gcluster explain modules/file-system/managed-lustre

# Or search the catalog directly by scheduler, machine type, storage, or module source:
grep -n "scheduler: slurm" skills/cluster-blueprint-assistant/references/catalog.yaml
grep -n "managed-lustre" skills/cluster-blueprint-assistant/references/catalog.yaml
```

Each `blueprints[]` entry gives you: `path`, optional `deployment_path`,
`scheduler`, `description`, `machine_types`, the default `vars`, and the
`modules` list with `id`, `source`, `use`, and configured `settings` keys.
Prefer blueprints under `examples/` over `community/examples/` when both match.

### 2. Customize via deployment.yaml

Always create a **new, untracked** deployment file (e.g. `my-deployment.yaml` in
the repo root or `examples/<name>-deployment.yaml`) containing only `vars:`.
**Never modify git-tracked companion `*-deployment.yaml` files** under
`examples/` or `community/examples/` in place (`deployment_path` in the catalog
entry, such as `examples/gke-a4/gke-a4-deployment.yaml` or
`examples/hpc-slurm-h4d/hpc-slurm-h4d-deployment.yaml`) — read them only to
inspect variable names, and omit `terraform_backend_defaults: type: gcs` unless
the user explicitly provides a GCS state bucket. Your new deployment file is
user-specific — do not commit it.

Only override keys that already exist in the blueprint's `vars:` (see the
catalog entry). Variable syntax is Cluster Toolkit's `$(vars.name)`, not
Terraform's `${var.name}`.

```yaml
# my-deployment.yaml
---
vars:
  project_id: my-proj
  deployment_name: hpc-slurm-small
  region: us-central1
  zone: us-central1-a
  compute_node_count_dynamic_max: 4
  compute_machine_type: c2-standard-30
```

> Global `vars:` are injected into **every** module that declares an input with
> the same name and does not set it explicitly. That is why blueprint vars use
> prefixed names such as `compute_machine_type` /
> `compute_node_count_dynamic_max`. Never add an unprefixed variable name that
> is shared by multiple modules in the blueprint: in `hpc-slurm.yaml`, for
> example, a bare `enable_spot_vm` would silently apply to all three nodesets
> (`debug_nodeset`, `compute_nodeset`, `h3_nodeset` — none of them sets it), and
> a bare `machine_type` would silently apply to `slurm_controller` only (the one
> module that does not set it explicitly). Where every module already sets the
> input explicitly (as `node_count_dynamic_max` is on all three nodesets in
> `hpc-slurm.yaml`), the bare name is injected nowhere and `gcluster` fails with
> `test_deployment_variable_not_used` instead.

#### Variable naming & override rules

Customizable vars in `examples/**/*.yaml` and `community/examples/**/*.yaml` follow
`<functional_role>_<exact_terraform_input>`; the suffix is the module input
the var feeds (`_machine_type`, `_node_count_dynamic_max`, `_node_count_static`,
`_static_node_count`, `_initial_node_count`, `_num_instances`, `_instance_count`,
`_autoscaling_total_min_nodes` / `_autoscaling_total_max_nodes`, `_size_gb`,
`_size_gib`, `_capacity_gb`, `_capacity_gib`, `_disk_size_gb`, `_disk_size`,
`_enable_spot_vm`, `_spot`), so no translation is needed between the var and
the setting. Every sizing, machine-type, storage-capacity and Spot knob on a
blueprint's worker compute and storage modules is exposed this way with a
concrete default equal to what the module would otherwise use (for example
`compute_enable_spot_vm: false` in `hpc-slurm.yaml`), so a user only ever
*has* to fill in `project_id`, `deployment_name`, `region` and `zone`.
Control-plane roles (`login_`, `controller_`, `system_node_pool_`) expose
machine type and disk size but never Spot: those modules do not accept it.

| Role prefix | Meaning |
| :--- | :--- |
| `compute_` | the worker nodes — a Slurm nodeset **or** a GKE node pool — when the blueprint has one |
| `login_` / `controller_` | the Slurm login / controller VM |
| `system_node_pool_` | the GKE system node pool (literal `gke-cluster` input) |
| `client_` | a plain VM that mounts storage (e.g. `pfs-managed-lustre-vm`) |
| `image_build_` | the temporary Packer VM that builds a custom image (`ml-slurm`, `ml-slurm-g4`) |
| `<partition>_` / `<pool>_` (`debug_`, `hp_pool_`, `local_ssd_pool_`, `n2_`, `h3_`, `a2_8_`) | one of several worker pools; the prefix is the partition / pool name the user already addresses |
| `homefs_`, `appsfs_`, `scratchfs_`, `sharefs_`, `lustre_`, `hyperdisk_<type>_`, `data_disk_` | storage, by mount role |

Rules the agent must respect:

- **Hardware-named partitions expose node counts and Spot, never
  `machine_type`** (`h3`, `a2`, `g2`, `n2`, `c2`, `c2d`, `c3`, `a2_8`,
  `a2_16`). The hardware name *is* the partition's identity; "run the `g2`
  partition on g4" is "add a partition" → Case 2/3 (copy the blueprint), not a
  `deployment.yaml` override. Functionally named pools (`compute`, `debug`,
  `hp_pool`, …) do expose `<role>_machine_type`.
- **Hardware-locked blueprints hardcode `machine_type` in the module.** The
  GKE and Slurm A3 / A4 / A4X blueprints have multi-NIC RDMA / GPUDirect wiring
  that only works with one shape, so they expose no compute machine-type var at
  all (the shape still appears in the catalog's `machine_types`). If the user
  asks for a different accelerator or machine series (e.g. `a3-highgpu-8g` on
  `gke-a4.yaml`), do **not** copy that blueprint and change `machine_type` —
  instead, use `./gcluster explain` to switch to the matching blueprint in
  `examples/` (e.g. `examples/gke-a3-highgpu/gke-a3-highgpu.yaml`). Multi-shape
  families (`gke-h4d`, `gke-tpu-*`, `gke-g4*`, all CPU blueprints) expose
  `compute_machine_type` with an `# Allowed:` comment listing valid shapes.
- **Three override states & GKE `SPECIFIC_RESERVATION`.** `""` means "empty
  string" (e.g. `compute_reservation_name: ""` in `ml-slurm-g4` or
  `h4d_reservation_name: ""` in `hpc-slurm-h4d` = no reservation); `$(null)`
  means "unset this input" (e.g. `static_node_count` must be unset under
  flex-start); a bare `key:` in YAML or `key=` on `--vars` is **always an
  error** (`deployment variable "key" was not set`). That error means a **null
  value in your `-d` file or `--vars`** — even though the location gcluster
  prints is the blueprint's `vars:` line, the fix belongs in `deployment.yaml`,
  not in the template. Note: if a GKE blueprint hardcodes
  `consume_reservation_type: SPECIFIC_RESERVATION` (such as `gke-tpu-v5e.yaml` or
  `gke-a4x.yaml`) and the user did not provide a
  reservation name, do **not** set `reservation: ""` (which generates an invalid
  empty `SPECIFIC_RESERVATION`); stop and ask whether the user has a reservation
  name or wants a custom blueprint copy (`ai_assisted: true`) configured for
  on-demand, Spot, or DWS Flex Start.
- **Precedence:** blueprint `vars:` < `-d deployment.yaml` < `--vars`. Later
  sources override earlier ones key by key.
- **Only use keys present in the catalog entry's `vars`, and never silence
  validators.** A `deployment.yaml` key that is *not* in the blueprint's `vars:`
  but happens to match a module input (a typo such as `machine_type` instead of
  `compute_machine_type`, or `enable_spot_vm` on a blueprint that has no Spot
  var) is **silently injected into every module that lacks that setting** — no
  error is raised as long as at least one module can absorb it (only when *no*
  module can does `test_deployment_variable_not_used` fail). Check `vars` in the
  catalog before writing the key. Never run `gcluster` with `-l IGNORE` /
  `--validation-level IGNORE`, never pass `--skip-validators` (except for the
  documented offline `GHPC_MOCK_MACHINE_CONFIG` dry-run when unauthenticated),
  and never add `validation_level: IGNORE` or `skip: true` under `validators:`
  in a blueprint YAML to silence a failing validator (`test_machine_type_in_zone`,
  `test_zone_in_region`, `test_deployment_variable_not_used`, etc.).
- **The catalog is the contract, not this document.** Official `examples/`
  follow the `<role>_<terraform_input>` convention; `community/examples/` may
  still use older names (`cluster_size`, `size_gib`, bare `machine_type`, …).
  Do not memorise exceptions: always read the chosen blueprint's `vars` list in
  `catalog.yaml` before writing keys to `deployment.yaml` or `--vars`. If a
  machine type appears in the catalog's `machine_types` but the entry has no
  `*_machine_type` var, it is hardware-locked — do not try to override it.
- Companion `*-deployment.yaml` files whose blueprint exposes prefixed vars end
  with a commented `Optional overrides (defaults shown)` block listing them;
  copy those variable names into your own untracked `deployment.yaml` rather than
  editing the tracked companion file in place.

If the blueprint does not expose the setting you need in `vars:`, copy the
tracked template to a new file in the same directory with `ai_assisted: true`
(or edit your existing custom copy in place), add a new `vars:` key with the
current hardcoded value as its default, reference it from the module via
`$(vars.<name>)`, and put the user's override value in `deployment.yaml` — this
keeps the existing behaviour and leaves tracked templates untouched.

### 3. Validate

```bash
# Dry-run (no deployment folder or expanded.yaml written; still runs cloud validators if authenticated):
./gcluster expand examples/hpc-slurm.yaml -d my-deployment.yaml -o /dev/null

# Offline (when using a placeholder project_id or without GCP credentials):
GHPC_MOCK_MACHINE_CONFIG='{}' ./gcluster expand examples/hpc-slurm.yaml -d my-deployment.yaml -o /dev/null \
  --skip-validators="test_project_exists,test_apis_enabled,test_region_exists,test_zone_exists,test_zone_in_region,test_quota_availability,test_machine_type_in_zone,test_reservation_exists,test_disk_type_in_zone"

# Full: writes the deployment folder and runs read-only cloud validators
# (project, APIs, zone, machine type, reservations). Use -w on re-runs instead
# of deleting the folder.
./gcluster create examples/hpc-slurm.yaml -d my-deployment.yaml -w
```

Read errors bottom-up. A path such as
`deployment_groups[0].modules[3].settings.machine_type` points at the blueprint
YAML — fix it there.

### 4. Common failures and where to fix them

| Symptom | Fix location |
| :--- | :--- |
| `a required setting "X" is missing from a module "Y"` | Add the module that outputs `X` to `Y`'s `use:` list (see `module_wiring.<source>.recommended_use` in the catalog, or the `outputs.tf` of candidate modules). |
| `module "X" was not used` | Add `X` to the `use:` list of its consumer (e.g. `private_service_access` → filestore / lustre). |
| `module "X" references unknown global variable "Y"` | Add `Y` under `vars:` in the blueprint or `deployment.yaml`. |
| `deployment variable "X" was not set` | A null value (`X:` with nothing after it) in **your** `-d` file or `--vars`, even though the location printed is the blueprint's `vars:` line. Set the value in `deployment.yaml`; do not edit the template. |
| `test_deployment_variable_not_used` / `the variable "X" was not used in this blueprint` | `X` is not a key in the blueprint's `vars:` and no module declares an input named `X` — almost always a typo'd key in `deployment.yaml`. Fix the key; never silence it with `-l IGNORE` or `--skip-validators`. |
| Reservation / machine type not found in zone | Change `zone` / `machine_type` in `vars:` or `deployment.yaml`. |
| Terraform error pointing at `<deployment_name>/<group>/main.tf` | Do **not** edit that file. Fix the blueprint and re-run `gcluster create ... -w` (or `gcluster deploy <blueprint> -d <deployment.yaml> -w` after confirmation). |

### 5. Deploy (gated)

Present this and wait for explicit confirmation before running anything that
creates cloud resources:

```markdown
[PROPOSED REMEDIATION PLAN]
- Target Resource: deployment <deployment_name> in project <project_id>
- Root Cause Identified: <why this deployment / change is needed>
- Proposed Action / Command: gcluster deploy examples/<blueprint>.yaml -d my-deployment.yaml
- Blast Radius: creates VPC, Filestore, Slurm controller/login/compute VMs (billable)
- Confirmation Required: Reply 'yes' to proceed.
```

## Out of scope

Debugging running VMs, Slurm jobs or GKE pods after deployment — use the
scheduler-specific skills (e.g. `gke-kueue-debugging`) for that.

