k8s-launch-kit-config

name	k8s-launch-kit-config
version	1.2.4
description	Use this skill when the user needs help understanding, creating, or editing a k8s-launch-kit (l8k) configuration file (l8k-config.yaml or cluster-config.yaml). Activate for: config file questions, parameter tuning, subnet configuration, NV-IPAM setup, DOCA driver settings, maintenance concurrency, NIC configuration operator settings, changing MTU, VFs, resource names, or understanding what any config field does.
metadata	{"requires":{"skills":["k8s-launch-kit-shared"]}}

l8k: Configuration Files

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and output modes.

Understand, create, or edit l8k configuration files.

File Types

File	Source	Purpose
cluster-config.yaml	Generated by l8k discover	Hardware inventory plus the resolved deployment profile
l8k-config.yaml	User-created or copied from cluster-config	Full config with both hardware + deployment settings

Usage

# Use a config file for generation
l8k generate --user-config my-config.yaml \
  --save-deployment-files ./output

# Combine user config with live discovery (hardware refreshed, settings kept)
l8k discover --user-config my-config.yaml \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./updated-config.yaml

Profile Resolution and Write-Back

Discovery and file-backed generation resolve and persist settings with this precedence:

1. Hardware and built-in defaults fill missing fields.
2. Values already present in --user-config are preserved.
3. Explicit CLI flags override both.

Discovery writes to its selected output YAML; generation rewrites its source config in place. On later runs, only missing fields are recalculated. multirail: false is an explicit value, not a missing field, so it remains false across rewrites.

Config Sections Quick Reference

Section	What It Controls
networkOperator	Operator namespace, version, image repository, helm chart repository (helmRepoURL)
docaDriver	OFED/DOCA driver image, version, blacklist settings
maintenance	Maintenance Operator, SR-IOV drain, and legacy OFED upgrade concurrency
nvIpam	NV-IPAM IP pool ranges and subnet generation
sriov	VF count, resource prefix, MTU, link type
hostdev	Host device resource name
rdmaShared	RDMA shared device resource name
ipoib	IPoIB master interface, resource name
macvlan	MacVLAN master interface, mode
nicConfigurationOperator	NIC firmware template settings
spectrumX	OVS bridge config, multiplane mode, RDMA settings
profile	Profile selection criteria (fabric, deployment, multirail)
clusterConfig[]	Per-group hardware: NICs, nodes, capabilities, selectors

Each clusterConfig[] entry has these key fields:

• identifier — group name (used for NicNodePolicy naming). For groups with both machineType and gpuType resolved, this is the lowercased machine/GPU identity with complete NVIDIA segments removed and common machine segments shortened (ThinkSystem → ts, PowerEdge → pe), bounded to 30 bytes with balanced component prefixes and a 6-character deterministic hash when needed; otherwise a fallback group-N. The Launch Kit machine node label uses the same value.
• machineType — server model (e.g. PowerEdge-XE9680); populated from nvidia.com/gpu.machine label or DMI fallback.
• gpuType — GPU SKU (e.g. NVIDIA-H200); populated from nvidia.com/gpu.product label or nvidia-smi fallback. Note: this field used to be called productType — the rename happened to disambiguate it from the server model. Old productType: keys in hand-authored configs must be renamed to gpuType:.
• capabilities.nodes.{sriov,rdma,ib} — what the underlying hardware supports.
• pfs[] — physical function list with PCI address, device ID, RDMA device, network interface, traffic class, rail, NUMA, GPU affinity, and model (the VPD model/description string read from NicDevice.Status.modelName).
  • One rail per NIC (default). Discovery advertises one rail per physical NIC: a NIC's multi-plane east-west PFs (planes of one port, e.g. Spectrum-X ConnectX-8/9) collapse to the master PF, so an 8-PF node lists 4 east-west PFs / 4 rails. A NIC whose model is genuinely dual-port (2-port/Dual-port) keeps a rail per port. Run l8k discover --collapse-nic-rails=false to emit one rail per PF (legacy/dev behaviour).
• nodeSelector — Kubernetes node selector for this group. Source groups key on the machine label written by l8k discover: nvidia.kubernetes-launch-kit.machine: <identifier>. Auto-merged groups (different machineTypes sharing a GPU type) key on nvidia.kubernetes-launch-kit.gpu: <gpuType> instead — the GPU label retains its discovered value, including , so the merged selector binds correctly across source machineTypes.

For the full field-by-field reference with types, defaults, and descriptions, read references/config-reference.md.

Topology Presets

The presets/ directory contains pre-recorded topologies for known hardware combinations. A preset is a topology.yaml file with the following shape:

machineType: PowerEdge-XE9680   # required
gpuType: NVIDIA-H200            # required — both keys are matched as a pair
nicModel: BlueField-3 SuperNIC (ConnectX-7)
gpuInterconnect: NV18
numaNodes: 2

# Required if the preset will be used with `l8k generate --for`.
# Discovery-time overlay does not need this block.
capabilities:
  nodes:
    sriov: true
    rdma: true
    ib: false

pfs:
  - deviceID: a2dc
    pciAddress: 0000:1a:00.0
    traffic: east-west
    rail: 0
    numaNode: 0
    connectedGPU: GPU0
    gpuProximity: PIX

Lookup is exact-match on (machineType, gpuType). No any-GPU fallback — a preset that doesn't declare gpuType: is rejected at load time. Multi-variant presets for the same machine (different GPU SKUs) live in separate directories with composite names like PowerEdge-XE9680-H200 / PowerEdge-XE9680-B200. The directory name is shown by l8k preset list and is what l8k generate --for <name> accepts.

Validation deviations. When the matched preset's PF count, PCI addresses, or device IDs don't exactly match discovered hardware, the preset is NOT applied — discovery keeps the live-discovered topology (traffic/rail/NUMA), because overlaying a preset onto a different PCI layout would corrupt the classification (a coincidentally-overlapping PCI address would inherit the preset's unrelated traffic/rail/GPU fields). The discrepancies are recorded under clusterConfig[*].presetDeviation and every subsequent config load re-emits a warning listing each deviation. The preset's authoritative topology is overlaid only on an exact match (zero deviations); presetApplied: true appears only in that case. Part-number and PSID differences are expected (firmware/SKU variants) and never block application.

Common Edits

# Change VF count per PF
sriov:
  numVfs: 16

# Change MTU
sriov:
  mtu: 9000

# Set DOCA driver version
docaDriver:
  version: "doca3.3.0-26.01-1.0.0.0-6"

# Allow four simultaneous maintenance operations. Network Operator 26.1+
# uses the global Maintenance Operator limits; older releases use the legacy
# SR-IOV/OFED limits where applicable.
maintenance:
  maxParallelOperations: 4
  maxUnavailable: 4
  maxNodeMaintenanceTimeSeconds: 3600
  maxParallelUpgrades: 4

# Override the helm chart repository URL (rarely needed — the embedded
# release catalog supplies the right URL for each MAJOR.MINOR release).
# Useful only for mirrors or private chart hosts.
networkOperator:
  helmRepoURL: "https://my-mirror.example.com/charts"

# Configure NV-IPAM subnets manually
nvIpam:
  subnets:
    - name: "rail-0-subnet"
      cidr: "10.10.0.0/16"
      gateway: "10.10.0.1"

# Namespaces for the secondary-network CRs + example test DaemonSets.
# One independent copy is rendered per namespace (shared resources like
# IPPools and NodePolicies are NOT duplicated). Defaults to ["default"].
networkNamespaces: []

Tips

• Start by running discovery (l8k discover) to generate a baseline, then edit it.
• A discovered config can be passed directly to l8k generate without repeating profile flags; use flags only for overrides.
• Use l8k schema to discover the Network Operator release keys supported by the installed l8k version.
• nvIpam subnets are auto-generated if not specified — one per rail using non-routable ranges.
• docaDriver.unloadThirdPartyRDMAModules: true auto-populates UNLOAD_THIRD_PARTY_RDMA_MODULES from discovered OFED-dependent modules.
• For release 26.1+, SR-IOV requestor mode requires both the Network Operator drain requestor and the SR-IOV external drainer. l8k renders both; applying only CRs cannot enable their Deployment environment variables.
• Updating an existing release to the generated requestor-mode Helm values requires --overwrite-existing.

See Also

• k8s-launch-kit-shared — Global flags
• k8s-launch-kit-discover — Generate a config from live cluster
• references/config-reference.md — Complete annotated YAML, including maintenance value restrictions