cluster-blueprint-assistant

# Cluster Toolkit Blueprint Assistant > [!WARNING] > This skill is experimental. Always notify the user before executing diagnostic steps and explicitly request confirmation for any proposed remediations. ## Core rules (read before touching any YAML) 1. **Start from `./gcluster explain` or the catalog, never from scratch.** Run `./gcluster explain <specs...>` (e.g. `./gcluster explain slurm a3u lustre`) or read [`references/catalog.yaml`](references/catalog.yaml) to find an existing blueprint under `examples/` or `community/examples/` that already matches the request. Do not run `ls`, `find`, or `grep` across `examples/` or `.tf` files to discover templates or modules; run `./gcluster explain <module>` (e.g. `./gcluster explain modules/file-system/managed-lustre`) or check the catalog's `module_wiring` section for a module's `required_inputs`, which module sources satisfy them (`recommended_use`), and which modules consume its outputs (`consumed_by`). If the request is a question ("do we have…", "can I get…", "what's in…"), answer it using `./gcluster explain` / `catalog.yaml` and confirm the approach with the user **before** editing anything. 2. **Customize through a new, untracked `deployment.yaml`, not module blocks.** If the request only changes project, region/zone, node counts, machine types, Spot/reservation, or storage size: - **Case 1 — All requested knobs already exist in `vars:`:** Create or edit **only** a **new, untracked** `deployment.yaml` (or `<name>-deployment.yaml`) passed with `-d`, containing only `vars:` (omit `terraform_backend_defaults: type: gcs` unless the user provides a GCS state bucket). Do **not** copy or edit the blueprint `.yaml` file, and **never modify git-tracked companion `*-deployment.yaml` files** under `examples/` or `community/examples/` in place (read them only as a reference for variable names). - **Case 2 — Promoting a hardcoded module setting to `vars:`:** When a requested setting is hardcoded inside a module block and missing from `vars:`, never modify a tracked template under `examples/` or `community/examples/` in place. Instead, copy the template to a new file in the **same directory** with `ai_assisted: true` (or reuse an existing custom copy in place), add the missing knob to `vars:` with a default **equal to the value that was hardcoded**, reference it in the module via `$(vars.<name>)`, and put the user's override value in `deployment.yaml`. - **A template only exposes knobs it already set explicitly.** Tracked blueprints promote to `vars:` only settings that were literal values in a module block; they do not add Spot, reservation, DWS or any other input the module block was silent about. If the knob you need is not in `vars:` (check the catalog entry), it is Case 2 — copy the blueprint. Never add the knob to the tracked template, and never pass it as a bare module-input name in `deployment.yaml` hoping it will be picked up. - **Mutually exclusive consumption models (Spot vs. DWS Flex Start vs. Reservations):** Spot VMs (`spot` / `enable_spot_vm`), DWS Flex Start (`enable_flex_start` / `dws_flex`), and Reservations (`reservation` / `SPECIFIC_RESERVATION`) cannot be combined on the same node pool or nodeset. If the user asks to enable Spot on a DWS Flex Start or Reservation blueprint (or vice versa), **never silently disable** the blueprint's existing consumption model (e.g. do not set `enable_flex_start: false` unprompted); stop, explain that the two models are mutually exclusive, and ask the user which single consumption model they want. - On casual or vague requests (e.g. a "small" or "cheap" cluster with Spot VMs and N nodes), override **only** the requested Spot and node-count variables (such as `compute_enable_spot_vm: true`, `debug_enable_spot_vm: true`, `h3_enable_spot_vm: true`, `compute_node_count_dynamic_max: 2`) in `deployment.yaml`—keep the blueprint's default machine types (never downgrade `c2-standard-60` to `e2-standard-2`, which fails `gvnic_enabled` validation) and never remove partitions or storage modules unprompted. - Do **not** add, remove, rename or restructure `deployment_groups[].modules[]` entries for value overrides. 3. **Only edit module blocks when adding or removing capability** (Case 3 — e.g. adding a Managed Lustre instance or a second node pool) — and do it on a **copy**, not the shared template: copy the blueprint to a new file in the **same directory** (e.g. `examples/gke-a4/my-gke-a4.yaml`) or copy the whole folder, because blueprints reference sibling files via `$(ghpc_stage("./…"))`. Never modify tracked template files under `examples/` or `community/examples/` in place (once a custom copy exists—identifiable by `ai_assisted: true` or untracked `git status`—edit that copy directly in place for follow-ups rather than copying it a second time). Copy the module block from `./gcluster explain <module>` or a blueprint in the catalog that already has it, keep its `use:` list, and check `module_wiring.<source>.required_inputs` to confirm every required input is satisfied by a module in `use:`. 4. **Mark agent-authored blueprints.** Add `ai_assisted: true` at the top level of any blueprint you create, copy, or repair/fix (if an existing custom blueprint you are fixing lacks `ai_assisted: true`, add it). 5. **Validate with gcluster, fix the blueprint, never the output.** `./gcluster expand` and `./gcluster create` are the validation step. Once `./gcluster create` or `./gcluster expand` succeeds, **stop immediately**—do not run `terraform init` or `terraform validate` afterwards. If they or `./gcluster deploy` fail, fix the blueprint YAML or `deployment.yaml` and re-run `./gcluster`. Never bypass validators via CLI flags (`-l IGNORE`, `--validation-level IGNORE`, `--skip-validators`), environment variables (`GHPC_MOCK_MACHINE_CONFIG`), or YAML fields (`validation_level: IGNORE`, or `skip: true` under `validators:`), **even if the user explicitly asks to skip a validator by name**—refuse the bypass, explain that skipping `test_machine_type_in_zone` still fails with a 404 Compute API error during module expansion (for Slurm `machine_configs` lookup) or deployment when the machine type does not exist in that zone, and ask the user for a supported zone or machine type. Never edit generated files under `<deployment_name>/` (`main.tf`, `variables.tf`, `terraform.tfvars`, `defaults.auto.pkrvars.hcl`, `.ghpc/`) and never run `terraform` in any form — not `init`, `validate`, `plan`, `apply`, `destroy`, or `fmt` — even if the user asks you to; refuse, explain, and offer the gcluster equivalent. `./gcluster` has no dry-run `plan` command: if asked for a dry-run plan or resource preview before deploying, never run `terraform plan` or pipe `yes n | ./gcluster deploy`; instead run `./gcluster explain <blueprint.yaml>` and inspect `<deployment_name>/.ghpc/artifacts/expanded_blueprint.yaml`. gcluster regenerates deployment files on every run. Generated `.tf`, `.tfvars` and `.pkrvars.hcl` files have a `# DO NOT EDIT: Generated by gcluster.` line just below the license header — if you see that line, you are in the wrong file, even if a terraform error points at it. A failed `./gcluster deploy` prints a `Hint:` with the exact `create … -w` / `deploy … -w` command to re-run after fixing the source. If you cannot reproduce a reported error with `./gcluster create`, stop and ask the user for the full error output instead of searching further. 6. **Destructive or mutating actions require confirmation.** `./gcluster deploy`, `./gcluster destroy`, and anything that creates or deletes cloud resources must be presented as a `[PROPOSED REMEDIATION PLAN]` and wait for the user to reply `yes`. The same applies to local destructive steps: deleting a deployment folder (`rm -rf <deployment_name>`), deleting or overwriting user `.yaml` files, and `git checkout` / `git restore` / `git reset` / `git clean` of user edits — present a `[PROPOSED REMEDIATION PLAN]` stating exactly what will be removed and wait for `yes`. Prefer `./gcluster create … -w` over deleting the deployment folder. 7. **Do only what was asked.** If you notice unrelated warnings or problems (deprecation notices, bugs in Go code, other blueprints), report them in your summary — do not fix, migrate or refactor them unprompted. ## Workflow ### 0. Ensure `./gcluster` is built From the root of the repository, verify that `./gcluster` exists or build it via `make`: ```bash test -x ./gcluster || make ``` ### 1. Find the right template Use `./gcluster explain` or search `references/catalog.yaml`: ```bash # Find the exact or closest blueprint for given specs (plus missing module snippets): ./gcluster explain slurm a3u lustre ./gcluster explain gke a3-megagpu lustre # Inspect a specific blueprint's required GCP APIs, customizable vars, and stages: ./gcluster explain examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml # Inspect a module's required APIs, auto-injected inputs, and ready-to-copy 'use:' snippet: ./gcluster explain modules/file-system/managed-lustre # Or search the catalog directly by scheduler, machine type, storage, or module source: grep -n "scheduler: slurm" skills/cluster-blueprint-assistant/references/catalog.yaml grep -n "managed-lustre" skills/cluster-blueprint-assistant/references/catalog.yaml ``` Each `blueprints[]` entry gives you: `path`, optional `deployment_path`, `scheduler`, `description`, `machine_types`, the default `vars`, and the `modules` list with `id`, `source`, `use`, and configured `settings` keys. Prefer blueprints under `examples/` over `community/examples/` when both match. ### 2. Customize via deployment.yaml Always create a **new, untracked** deployment file (e.g. `my-deployment.yaml` in the repo root or `examples/<name>-deployment.yaml`) containing only `vars:`. **Never modify git-tracked companion `*-deployment.yaml` files** under `examples/` or `community/examples/` in place (`deployment_path` in the catalog entry, such as `examples/gke-a4/gke-a4-deployment.yaml` or `examples/hpc-slurm-h4d/hpc-slurm-h4d-deployment.yaml`) — read them only to inspect variable names, and omit `terraform_backend_defaults: type: gcs` unless the user explicitly provides a GCS state bucket. Your new deployment file is user-specific — do not commit it. Only override keys that already exist in the blueprint's `vars:` (see the catalog entry). Variable syntax is Cluster Toolkit's `$(vars.name)`, not Terraform's `${var.name}`. ```yaml # my-deployment.yaml --- vars: project_id: my-proj deployment_name: hpc-slurm-small region: us-central1 zone: us-central1-a compute_node_count_dynamic_max: 4 compute_machine_type: c2-standard-30 ``` > Global `vars:` are injected into **every** module that declares an input with > the same name and does not set it explicitly. That is why blueprint vars use > prefixed names such as `compute_machine_type` / > `compute_node_count_dynamic_max`. Never add an unprefixed variable name that > is shared by multiple modules in the blueprint: in `hpc-slurm.yaml`, for > example, a bare `enable_spot_vm` would silently apply to all three nodesets > (`debug_nodeset`, `compute_nodeset`, `h3_nodeset` — none of them sets it), and > a bare `machine_type` would silently apply to `slurm_controller` only (the one > module that does not set it explicitly). Where every module already sets the > input explicitly (as `node_count_dynamic_max` is on all three nodesets in > `hpc-slurm.yaml`), the bare name is injected nowhere and `gcluster` fails with > `test_deployment_variable_not_used` instead. #### Variable naming & override rules Customizable vars in `examples/**/*.yaml` and `community/examples/**/*.yaml` follow `<functional_role>_<exact_terraform_input>`; the suffix is the module input the var feeds (`_machine_type`, `_node_count_dynamic_max`, `_node_count_static`, `_static_node_count`, `_initial_node_count`, `_num_instances`, `_instance_count`, `_autoscaling_total_min_nodes` / `_autoscaling_total_max_nodes`, `_size_gb`, `_size_gib`, `_capacity_gb`, `_capacity_gib`, `_disk_size_gb`, `_disk_size`, `_enable_spot_vm`, `_spot`), so no translation is needed between the var and the setting. Every sizing, machine-type, storage-capacity and Spot knob on a blueprint's worker compute and storage modules is exposed this way with a concrete default equal to what the module would otherwise use (for example `compute_enable_spot_vm: false` in `hpc-slurm.yaml`), so a user only ever *has* to fill in `project_id`, `deployment_name`, `region` and `zone`. Control-plane roles (`login_`, `controller_`, `system_node_pool_`) expose machine type and disk size but never Spot: those modules do not accept it. | Role prefix | Meaning | | :--- | :--- | | `compute_` | the worker nodes — a Slurm nodeset **or** a GKE node pool — when the blueprint has one | | `login_` / `controller_` | the Slurm login / controller VM | | `system_node_pool_` | the GKE system node pool (literal `gke-cluster` input) | | `client_` | a plain VM that mounts storage (e.g. `pfs-managed-lustre-vm`) | | `image_build_` | the temporary Packer VM that builds a custom image (`ml-slurm`, `ml-slurm-g4`) | | `<partition>_` / `<pool>_` (`debug_`, `hp_pool_`, `local_ssd_pool_`, `n2_`, `h3_`, `a2_8_`) | one of several worker pools; the prefix is the partition / pool name the user already addresses | | `homefs_`, `appsfs_`, `scratchfs_`, `sharefs_`, `lustre_`, `hyperdisk_<type>_`, `data_disk_` | storage, by mount role | Rules the agent must respect: - **Hardware-named partitions expose node counts and Spot, never `machine_type`** (`h3`, `a2`, `g2`, `n2`, `c2`, `c2d`, `c3`, `a2_8`, `a2_16`). The hardware name *is* the partition's identity; "run the `g2` partition on g4" is "add a partition" → Case 2/3 (copy the blueprint), not a `deployment.yaml` override. Functionally named pools (`compute`, `debug`, `hp_pool`, …) do expose `<role>_machine_type`. - **Hardware-locked blueprints hardcode `machine_type` in the module.** The GKE and Slurm A3 / A4 / A4X blueprints have multi-NIC RDMA / GPUDirect wiring that only works with one shape, so they expose no compute machine-type var at all (the shape still appears in the catalog's `machine_types`). If the user asks for a different accelerator or machine series (e.g. `a3-highgpu-8g` on `gke-a4.yaml`), do **not** copy that blueprint and change `machine_type` — instead, use `./gcluster explain` to switch to the matching blueprint in `examples/` (e.g. `examples/gke-a3-highgpu/gke-a3-highgpu.yaml`). Multi-shape