k8s-launch-kit-config

# l8k: Configuration Files > **PREREQUISITE:** Read `../k8s-launch-kit-shared/SKILL.md` for install paths, global flags, and output modes. Understand, create, or edit l8k configuration files. ## File Types | File | Source | Purpose | |------|--------|---------| | `cluster-config.yaml` | Generated by `l8k discover` | Hardware inventory plus the resolved deployment profile | | `l8k-config.yaml` | User-created or copied from cluster-config | Full config with both hardware + deployment settings | ## Usage ```bash # Use a config file for generation l8k generate --user-config my-config.yaml \ --save-deployment-files ./output # Combine user config with live discovery (hardware refreshed, settings kept) l8k discover --user-config my-config.yaml \ --kubeconfig ~/.kube/config \ --save-cluster-config ./updated-config.yaml ``` ## Profile Resolution Fresh discovery and file-backed generation resolve settings with this precedence: 1. Canonical built-in defaults from the repository-root `cluster-config.yaml`. 2. Hardware-derived profile defaults. 3. Existing config values. 4. Explicit CLI flags. Generation never rewrites the source config. It stores the exact effective configuration at `<output>/.l8k/resolved-config.yaml`; deploy and validate use that sidecar automatically when no explicit config is passed. Explicit false, zero, and empty values override defaults. YAML `null` means unset. Discovery with `--user-config` follows a stricter refresh contract: only `clusterConfig` is replaced. Every other section remains as loaded unless an explicit CLI flag overrides its corresponding field. Generation still fills missing profile fields when it consumes that config. ## Config Sections Quick Reference | Section | What It Controls | |---------|-----------------| | `networkOperator` | Operator namespace, version, image repository, Helm repository, and `skipHelmChart` ownership switch | | `docaDriver` | OFED/DOCA driver image, version, blacklist settings | | `maintenance` | Maintenance Operator, SR-IOV drain, and legacy OFED upgrade concurrency | | `nvIpam` | NV-IPAM per-node block size, IP pool ranges, and subnet generation | | `sriov` | VF count, resource prefix, MTU, link type | | `hostdev` | Host device resource name | | `rdmaShared` | RDMA shared device resource name | | `ipoib` | IPoIB master interface, resource name | | `macvlan` | MacVLAN master interface, mode | | `nicConfigurationOperator` | NIC firmware template settings | | `spectrumX` | OVS bridge config, multiplane mode, RDMA settings | | `profile` | Profile selection criteria (fabric, deployment, multirail) | | `clusterConfig[]` | Per-group hardware: NICs, nodes, capabilities, selectors | Each `clusterConfig[]` entry has these key fields: - `identifier` — group name (used for `NicNodePolicy` naming). For groups with both `machineType` and `gpuType` resolved, this is the lowercased machine/GPU identity with complete `NVIDIA` segments removed and common machine segments shortened (`ThinkSystem` → `ts`, `PowerEdge` → `pe`), bounded to 30 bytes with balanced component prefixes and a 6-character deterministic hash when needed; otherwise a fallback `group-N`. The Launch Kit machine node label uses the same value. - `machineType` — server model (e.g. `PowerEdge-XE9680`); populated from `nvidia.com/gpu.machine` label or DMI fallback. - `gpuType` — GPU SKU (e.g. `NVIDIA-H200`); populated from `nvidia.com/gpu.product` label or `nvidia-smi` fallback. **Note:** this field used to be called `productType` — the rename happened to disambiguate it from the server model. Old `productType:` keys in hand-authored configs must be renamed to `gpuType:`. - `netplanManaged` — true when any worker has an NVIDIA PF whose current MAC is selected by a host netplan `match.macaddress` stanza with a non-empty `set-name`. The flag identifies a potential conflict with NCO udev naming; host-specific MACs are not persisted. If generation would emit a `NicInterfaceNameTemplate` for this group, clean up the affected `set-name` stanzas and re-run discovery instead of editing this flag by hand. - `capabilities.nodes.{sriov,rdma,ib}` — what the underlying hardware supports. - `pfs[]` — physical function list with PCI address, device ID, RDMA device, network interface, traffic class, rail, NUMA, GPU affinity, and `model` (the VPD model/description string read from `NicDevice.Status.modelName`). - **One rail per NIC (default).** Discovery advertises one rail per physical NIC: a NIC's multi-plane east-west PFs (planes of one port, e.g. Spectrum-X ConnectX-8/9) collapse to the master PF, so an 8-PF node lists 4 east-west PFs / 4 rails. A NIC whose `model` is genuinely dual-port (`2-port`/`Dual-port`) keeps a rail per port. Run `l8k discover --collapse-nic-rails=false` to emit one rail per PF (legacy/dev behaviour). - `nodeSelector` — Kubernetes node selector for this group. Source groups key on the machine label written by `l8k discover`: `nvidia.kubernetes-launch-kit.machine: <identifier>`. Auto-merged groups (different machineTypes sharing a GPU type) key on `nvidia.kubernetes-launch-kit.gpu: <gpuType>` instead — the GPU label retains its discovered value, including `NVIDIA`, so the merged selector binds correctly across source machineTypes. - `workerNodes` — explicit hostnames (populated by discovery). For the full field-by-field reference with types, defaults, and descriptions, read `references/config-reference.md`. ## Topology Presets The `presets/` directory contains pre-recorded topologies for known hardware combinations. A preset is a `topology.yaml` file with the following shape: ```yaml machineType: PowerEdge-XE9680 # required gpuType: NVIDIA-H200 # required — both keys are matched as a pair nicModel: BlueField-3 SuperNIC (ConnectX-7) gpuInterconnect: NV18 numaNodes: 2 # Required if the preset will be used with `l8k generate --for`. # Discovery-time overlay does not need this block. capabilities: nodes: sriov: true rdma: true ib: false pfs: - deviceID: a2dc pciAddress: 0000:1a:00.0 traffic: east-west rail: 0 numaNode: 0 connectedGPU: GPU0 gpuProximity: PIX ``` **Lookup is exact-match on `(machineType, gpuType)`.** No any-GPU fallback — a preset that doesn't declare `gpuType:` is rejected at load time. Multi-variant presets for the same machine (different GPU SKUs) live in separate directories with composite names like `PowerEdge-XE9680-H200` / `PowerEdge-XE9680-B200`. The directory name is shown by `l8k preset list` and is what `l8k generate --for <name>` accepts. **Validation deviations.** When the matched preset's PF count, PCI addresses, or device IDs don't exactly match discovered hardware, the preset is **NOT applied** — discovery keeps the live-discovered topology (traffic/rail/NUMA), because overlaying a preset onto a different PCI layout would corrupt the classification (a coincidentally-overlapping PCI address would inherit the preset's unrelated traffic/rail/GPU fields). The discrepancies are recorded under `clusterConfig[*].presetDeviation` and every subsequent config load re-emits a warning listing each deviation. The preset's authoritative topology is overlaid **only on an exact match** (zero deviations); `presetApplied: true` appears only in that case. Part-number and PSID differences are expected (firmware/SKU variants) and never block application. ## Common Edits ```yaml # Change VF count per PF and fabric-specific MTUs sriov: numVfs: 16 ethernetMtu: 9000 infinibandMtu: 4000 # Set the DOCA driver version. Advanced users can also forward literal # environment variables through the generated NicClusterPolicy or # NicNodePolicy; custom values override any value produced by another # docaDriver field with the same name. docaDriver: version: "doca3.3.0-26.01-1.0.0.0-6" env: - name: THIRD_PARTY_RDMA_MODULES value: "nvidia_peermem" # Allow four simultaneous maintenance operations. Network Operator 26.1+ # uses the global Maintenance Operator limits; older releases use the legacy # SR-IOV/OFED limits where applicable. maintenance: maxParallelOperations: 4 maxUnavailable: 4 maxNodeMaintenanceTimeSeconds: 3600 maxParallelUpgrades: 4 # The selected release supplies the Helm repository URL. networkOperator: # Keep generating/applying Network Operator CRs but let another system own # the Helm release. Generate/deploy/validate have the equivalent CLI flag # --skip-network-operator-helm; clean reads this persistent ownership setting # and retains the release while deleting Network Operator CRs. skipHelmChart: true # Configure NV-IPAM subnets manually nvIpam: perNodeBlockSize: 10 subnets: - subnet: "10.10.0.0/16" gateway: "10.10.0.1" # Namespaces for the secondary-network CRs + example test DaemonSets. # One independent copy is rendered per namespace (shared resources like # IPPools and NodePolicies are NOT duplicated). Defaults to ["default"]. networkNamespaces: ["my-namespace"] ``` ## Tips - Start by running discovery (`l8k discover`) to generate a baseline, then edit it. - A discovered config can be passed directly to `l8k generate` without repeating profile flags; use flags only for overrides. - Use `l8k schema` to discover the Network Operator release keys supported by the installed l8k version. - Treat a GA catalog entry as sticky during nightly synchronization: an upstream beta or release candidate for the next patch must not replace its public Network Operator artifact set. Update the entry again when the patch is GA. - `nvIpam` subnets are auto-generated if not specified — one per rail using non-routable ranges. - `nvIpam.perNodeBlockSize` defaults to `10`; generation warns when it is lower than `sriov.numVfs` because the per-node allocation may be too small for all VFs. - `docaDriver.unloadThirdPartyRDMAModules: true` auto-populates `UNLOAD_THIRD_PARTY_RDMA_MODULES` from discovered OFED-dependent modules. - `docaDriver.env` is an advanced escape hatch. Values override generated MOFED environment entries by name, duplicate custom names use the last value, and bad driver options can disrupt node networking. - For release 26.1+, SR-IOV requestor mode requires both the Network Operator drain requestor and the SR-IOV external drainer. l8k renders both; applying only CRs cannot enable their Deployment environment variables. - Updating an existing release to the generated requestor-mode Helm values requires `--overwrite-existing`. ## See Also - [k8s-launch-kit-shared](../k8s-launch-kit-shared/SKILL.md) — Global flags - [k8s-launch-kit-discover](../k8s-launch-kit-discover/SKILL.md) — Generate a config from live cluster - `references/config-reference.md` — Complete annotated YAML, including maintenance value restrictions ## OpenShift OpenShift configuration uses top-level `flavor: ocp`, `sriov.operatorNamespace`, `nfd.operatorNamespace`, `nfd.configurationName`, and `maintenance.operatorNamespace`. Install operator packages externally; l8k configures existing installations. ## Optional standard NIC tuning `nicConfigurationOperator.deployNicConfigurationTemplate` defaults to false. Root, generate and discover accept `--deploy-nic-configuration-template[=false]`. It enables NCO independently of interface naming and renders per-source, east-west PCI-scoped templates for standard SR-IOV, RDMA-shared and host-device profiles on k8s/OCP. VF count comes from `sriov.numVfs`; link type comes from `profile.fabric`. Ethernet gets DSCP trust and priority-3 PFC; InfiniBand omits RoCE tuning. Spectrum-X rejects this option. Only SR-IOV disables the Mellanox plugin (Helm values on k8s, additive `disablePlugins` merge on OCP). Kubernetes external Helm owners must configure that plugin themselves. Deploy and validate consume the generated template and require all matched NicDevices to report current `ConfigUpdateInProgress=False` with reason `UpdateSuccessful`; stale/partial status waits and failures surface. Opting out does not restore applied NIC settings. On k8s, omitted templates (also for unselected groups) block deploy preflight unless `--overwrite-existing` is used, which deletes them. OCP preserves existing templates and disabled plugins. Generation rejects inventory with selected and excluded ports on the same PCI device. Review whole-NIC scope, including ports absent from inventory, and qualify runtime RoCE support for the installed NCO/DOCA/RHCOS versions before live use. See `docs/reference/configuration.md#nic-configuration-and-naming`.