xpk-to-clustertoolkit

> [!WARNING] > This skill is experimental. Always review generated blueprints and command translations before execution, and explicitly request confirmation for any mutating deployments. <!-- pymarkdown:disable heading-style,single-title --> # Overview You are an expert in migrating `xpk` CLI configurations and commands to Cluster Toolkit (`gcluster`). Your goal is to automate the translation of `xpk` commands found in Markdown files and Bash scripts into their equivalent Cluster Toolkit representations. ## Canonical Documentation & Reference Tier For deep technical details and comprehensive translation matrices, consult the authoritative documentation: 1. **Migration Guide**: Read `docs/migration/xpk_to_clustertoolkit.md` (or online at [XPK to Cluster Toolkit Migration Guide](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/docs/migration/xpk_to_clustertoolkit.md)). 2. **Flag Mapping Matrix**: Read [flag_mappings.md](references/flag_mappings.md) for 1:1 flag translation and `--filter-by-status` to `--status` mappings. 3. **Architectural Differences**: Read [architectural_differences.md](references/architectural_differences.md) for IaC lifecycle, KubeRay, and storage comparisons. 4. **End-to-End Examples**: Read [examples.md](references/examples.md) for full worked migration recipes (MaxText RL, multi-slice TPU, NVIDIA GPU, Lustre). --- # Core Workflows When processing target files, stitch together line continuations (`\`) and evaluate unexpanded shell parameters. ## 1. 1:1 Mapping Rule (Simple Commands) For simple `xpk` commands, perform inline replacements: * `xpk workload list` -> `gcluster job list` (filters workloads via `--name-contains`). * `xpk workload delete <name>` -> `gcluster job cancel <name>` (positional workload name; batch deletion via `--filter-by-job` is not supported). * `xpk cluster delete` -> `gcluster destroy <deployment_directory>` (takes the path to the deployment directory, e.g. `./<deployment_name>`). * `xpk config set` -> `gcluster job config set` (Cluster Toolkit accepts only `project`, `cluster`, `location`). Setting this preamble in workflow scripts allows subsequent `job submit`, `job list`, and `job logs` commands to omit connection flags. * `xpk inspector` -> `gcluster job logs <workload_name>` (or `gcluster job inspect <workload_name>`). * `xpk info` -> `gcluster cluster info` * `xpk cluster list` -> `gcluster cluster list` * `xpk cluster describe` -> `gcluster cluster describe` * `xpk cluster adapt` -> Remove command (`# Prerequisite tools checked via gcluster; cluster CRDs deployed via blueprint`). * `xpk storage list` -> `gcluster cluster volume` ## 2. Complex Commands (`workload create` & `cluster create`) **MANDATORY RULE ON WORKLOAD TRANSLATION**: * **Always use `gcluster job submit`**: Translate all `xpk workload create` or `xpk workload create-pathways` commands to `gcluster job submit`. * **Avoid pure `kubectl` commands**: Do NOT generate raw Kubernetes manifests (e.g. raw `Job`, `JobSet`, or `Pod` YAMLs applied via `kubectl apply`) as replacements for `xpk workload create`, unless specifically migrating custom operators like KubeRay (`RayJob`). **Deterministic Parser Invocation**: Always run the parser with **single quotes** so shell variables (`$CLUSTER`, `$ZONE`) reach the parser unexpanded: ```bash python3 skills/xpk-to-clustertoolkit/scripts/parse_xpk_to_gcluster.py 'xpk ...' ``` *(Adjust the script path if vendored outside the repository root).* > [!IMPORTANT] > `parse_xpk_to_gcluster.py` is the authoritative source of truth for flag mappings. > Give precedence to user experience and correctness: never silently parse or drop unrecognized flags. > When the parser encounters an unmapped or unrecognized flag, it emits an explicit `# Warning: Unmapped xpk flags were ignored:` note clarifying that the tool is unsure how to translate it automatically. > When handling these flags: (a) clearly warn the user about any unrecognized flags and explain that the tool is unsure of their translation, (b) record them in the Migration Report's Unmapped Flags table with behavioral impacts, and (c) do NOT leave inline comments naming the legacy XPK flag in executable scripts. If the flag is load-bearing (`--force`, `--dry-run`), ask the user before proceeding. ### 2a. Workload Submission (`gcluster job submit`) * **Standard TPU Workloads**: Converted to `gcluster job submit --compute-type <type> --topology <AxBxC> --command <cmd>`. * **TPU `--num-nodes` Constraint**: `gcluster job submit` rejects `--num-nodes` for TPU jobs (`cmd/job/submit.go:226 (validateTPUFlags)`). The parser automatically omits `--num-nodes` for TPUs. * **GPU Workloads**: GPU jobs take `--num-nodes` and **never** `--topology`. The parser preserves `--num-nodes` for GPU compute types (e.g., `a3-megagpu-8g`, `h100-mega-80gb-8`). * **Pathways Workloads (`xpk workload create-pathways`)**: * Adds `--pathways` and maps `--headless` -> `--pathways-headless`. * `--pathways-gcs-location <gs-uri>` is mandatory in Cluster Toolkit. * **Workload Name Length**: Standard names must be <= 28 characters; Pathways names must be <= 22 characters due to Kubernetes 63-byte coordinator label limits (`<name>-pathways-head-0-0.<name>`). * **Multi-Tier Checkpointing (MTC)**: Maps `--mtc-enabled` -> `--gke-mtc-enabled`, `--ramdisk-directory` -> `--gke-mtc-ramdisk-dir`, and `--colocated-python-sidecar-image` -> `--pathways-colocated-python-sidecar-image`. (Cluster requires `HighScaleCheckpointing` and `GcsFuseCsiDriver` addons). * **Elastic Slices**: Maps `--elastic-slices` -> `--pathways-elastic-slices` and `--max-slice-restarts` -> `--pathways-max-slice-restarts`. * **Container Images & Crane Builds**: `--docker-image` designates the direct workload container image (`--image`). `--base-docker-image` paired with `--script-dir` maps to `--base-image` + `--build-context` for Crane builds. Note that xpk rejects combining `--docker-image` with `--base-docker-image` or `--script-dir`; following xpk precedence, `--docker-image` takes precedence and emits `--image` alone without build context. * **Storage Mounts**: In xpk, `--storage <name>` references a separate Storage object whose mount point, PVC name, and read-only mode live on the CRD rather than the CLI command. The parser emits a failsafe placeholder `--mount "<src>;/mnt/<src>;ro"` (defaulting to read-only `ro`, matching gcluster's default) with a warning to verify the real parameters via `xpk storage list` or `gcluster cluster volume`. Destination paths are automatically deduplicated (`/mnt/data`, `/mnt/data_2`). **Standard Workload Example:** ```bash # Input: xpk workload create --workload my-workload --tpu-type tpu7x-4x4x4 --command "python3 train.py" # Output: gcluster job submit --name my-workload --command 'python3 train.py' --compute-type tpu7x-standard-4t --topology 4x4x4 ``` **Pathways Workload Example:** ```bash # Input: xpk workload create-pathways --workload my-pw-job --tpu-type tpu7x-4x4x4 --headless --pathways-gcs-location gs://my-bucket/tmp --command "python3 train.py" # Output: gcluster job submit --pathways --name my-pw-job --command 'python3 train.py' --pathways-gcs-location gs://my-bucket/tmp --pathways-headless --compute-type tpu7x-standard-4t --topology 4x4x4 ``` ### 2b. Blueprint Generation (`cluster create`) For `xpk cluster create` and `create-pathways`, generate a declarative Cluster Toolkit blueprint: 1. **Asset Selection & Blueprint Copying**: Never modify files under `examples/` in place — they are shared, CI-tested assets. Copy the canonical example next to the source script and name it `<deployment_name>.yaml`; all edits go to the copy: * **TPU v4 / v5e / v5p / v6e**: `examples/gke-tpu-v6e/gke-tpu-v6e.yaml`, `examples/gke-tpu-v4/gke-tpu-v4.yaml` * **TPU 7x**: `examples/gke-tpu-7x/gke-tpu-7x.yaml` * **NVIDIA GPU (A4 / A3)**: `examples/gke-a4/gke-a4.yaml`, `examples/gke-a3-megagpu/gke-a3-megagpu.yaml` * **Managed Lustre**: `examples/gke-managed-lustre.yaml` * **Storage (GCS FUSE, Filestore, Persistent Volumes)**: `examples/storage-gke.yaml` *(In standalone environments where the repository is not cloned, run `skills/xpk-to-clustertoolkit/scripts/fetch_toolkit_examples.sh` to sync canonical examples to `~/.cache/cluster-toolkit-examples/`, or download directly via `curl https://raw.githubusercontent.com/GoogleCloudPlatform/cluster-toolkit/main/examples/<path>`).* 2. **Additive Merge Semantics**: Merging is **additive**: overwrite only the keys the parser emitted and preserve every other key already present in the example's `vars:` block. * **Pre-existing Unset Variables**: Canonical examples (like `examples/gke-tpu-v6e/gke-tpu-v6e.yaml`) declare `authorized_cidr` and `reservation` with no default value. Supply them via `--vars` or prompt the user. * **Reservation Affinity**: If `--reservation` is omitted in the XPK command, delete the pre-existing `reservation_affinity` block from the accelerator node pool module and the now-unused `reservation` variable from `vars:` (it is the block's only consumer in canonical blueprints; leaving it trips `test_deployment_variable_not_used`). * **GPU Blueprints**: Do NOT merge `compute_machine_type` into `examples/gke-a3-megagpu/` or `examples/gke-a4/` — those blueprints default it to the only machine type the node pool supports. Merge only `static_node_count`, `project_id`, `deployment_name`, `zone`, and `region`. 3. **Decision Rule for `vars:` vs `settings:`**: The parser emits three distinct configuration buckets: * **`vars:`**: Global blueprint variables (e.g. `project_id`, `deployment_name`, `reservation`, `enable_pathways_for_tpus`). In modern unified GPU blueprints (`gke-a3-megagpu`, and as of PRs #6276/#6279 `gke-a3-highgpu`, `gke-a3-ultragpu`, `gke-a4`), consumption models are configured directly in `vars:` (e.g. `vars.spot`, `vars.enable_flex_start`, and `vars.reservation_affinity`). * **`cluster_module_settings:`**: Module settings applied directly under `modules/scheduler/gke-cluster` `settings:` (e.g. `authorized_cidr`). * **`nodepool_module_settings:`**: Node pool settings applied directly under `modules/compute/gke-node-pool` `settings:` (e.g. `spot`, `enable_flex_start`, `auto_repair`, and for TPU/other blueprints, `reservation_affinity`). * *Reference Pattern*: When an example declares a key in `vars:` and references it in settings as `$(vars.X)` (e.g. `enable_pathways_for_tpus: $(vars.enable_pathways_for_tpus)` or `reservation_affinity.specific_reservations[0].name: $(vars.reservation)`), update the value in `vars:` and leave the reference intact. This keeps values `--vars`-settable at deploy time and prevents `test_deployment_variable_not_used` errors. * Do NOT add global variables that the example does not declare; explicit module settings block global propagation and trigger `test_deployment_variable_not_used`. 4. **Pathways Configuration**: * **When migrating with Pathways (`--enable-pathways`)**: Set `enable_pathways_for_tpus: true` in `vars:`. This automatically provisions the dedicated `cpu-np` node pool (`n4-standard-64`). * **When migrating WITHOUT Pathways**: You MUST explicitly set `enable_pathways_for_tpus: false` (emitted automatically by the parser for TPU clusters). Canonical blueprints (e.g. `gke-tpu-v6e.yaml:53 (enable_pathways_for_tpus)`) default it to `true` and will otherwise provision an unrequested CPU node pool. Do not set this variable on GPU blueprints. 5. **Reservation Affinity Targeting**: For modern unified GPU blueprints (`gke-a3-megagpu`, and as of PRs #6276/#6279 `gke-a3-highgpu`, `gke-a3-ultragpu`, `gke-a4`), configure the `reservation_affinity` object directly in `vars:`. For TPU and specialized blueprints (`gke-tpu-v6e-pool`, `gke-tpu-7x-pool`, `a4x_pool`), configure `reservation: <name>` in `vars:` and ensure the accelerator node pool module references `- name: $(vars.reservation)` under `settings:`. If the example already defines `reservation_affinity`, edit the existing block rather than adding a second one. 6. **Storage & Persistent Volume Architecture**: * A `modules/file-system/*` module provisions storage instances but creates **no** PVC. Always pair it with `modules/file-system/gke-persistent-volume` (`use: [<cluster>, <fs-module>]`). * In `examples/gke-managed-lustre.yaml`, the generated PersistentVolumeClaim claim name is **`gke-lustre-instance-pvc`** (not `lustre-pvc`). Workloads mount this claim directly. * `<src>` resolution: Cloud Storage -> `gs://bucket/path`; PVC provisioned by `gke-persistent-volume` -> bare claim name; Filestore -> `filestore://<instance>/<share>`. `<dest>` was set by `xpk storage create --mount-point` and cannot be derived from the submission command; ask the user rather than guessing. 7. **GHPC Template Expressions & HCL Arithmetic**: Template expansions inside `$(...)` evaluate as native HCL expressions. Compound arithmetic is fully supported (e.g. `$(vars.num_slices * pool.node_count_static * pool.tpu_chips_per_node)`). 8. **Deployment Command**: ```bash gcluster deploy <blueprint_file.yaml> --vars project_id=<project>,deployment_name=<name>,zone=<zone>,region=<region> ```