k8s-launch-kit-discover

# l8k: Cluster Discovery > **PREREQUISITE:** Read `../k8s-launch-kit-shared/SKILL.md` for install paths, global flags, and output modes. This workflow uses the default `host` target; `--target host` is equivalent. The CLI snapshots discovery arguments and runs the concrete Host discovery operation through the target registry; command syntax and artifacts are unchanged. Discover cluster hardware and produce a `cluster-config.yaml` describing NICs, GPUs, rails, and node groups. Use `docs/user/profiles.md#check-prerequisites-and-site-inputs` for the site-owned decisions and `docs/user/discovery.md#review-saved-inventory` for the post-discovery review before generation. Three things to know about the saved file: - **Fresh discovery persists a resolved profile.** Without `--user-config`, discovery fills profile fields from hardware/defaults and applies explicit CLI overrides last. With `--user-config`, it replaces only `clusterConfig`, preserves every other section, and then applies explicit CLI overrides. - **North-south-only groups are dropped.** A node group whose NICs are all north-south (BlueField DPUs / OOB NICs) produces no manifests and is omitted from `cluster-config.yaml` (it would otherwise consume an NV-IPAM subnet slice and shift the per-rail pools). North-south NICs alongside east-west NICs in a mixed group are still listed. - **Documentation comments are preserved.** The field-reference comments from the source config (`l8k-config.yaml` or `--user-config`) are carried over into the saved `cluster-config.yaml`, with a `Generated by l8k discover` banner at the top. **l8k discover is self-contained.** It does NOT require a pre-installed Network Operator. On every run it bootstraps the NIC Configuration Daemon (and the 5 nic-configuration-operator CRDs, only if missing) into a private namespace `nvidia-k8s-launch-kit`, reads `NicDevice` CRs published by the daemon, then tears the namespace down on exit (unless `--keep-namespace` is set). The daemon image is pulled from `<networkOperator.repository>/ nic-configuration-operator-daemon:<networkOperator.componentVersion>` (both fields come from `l8k-config.yaml`). ## Usage (from AI agent) ```bash l8k discover \ --kubeconfig ~/.kube/config \ --save-cluster-config ./cluster-config.yaml \ --output json >discovery.json 2>discovery.log && jq . discovery.json ``` Check the `l8k` exit status before parsing; retain `discovery.log` on failure. ## Usage (human-interactive) ```bash l8k discover --save-cluster-config <OUTPUT> [--kubeconfig <PATH>] ``` ## Flags | Flag | Required | Default | Description | |------|----------|---------|-------------| | `--kubeconfig` | — | `$KUBECONFIG` env var | Path to kubeconfig (optional — falls back to env var) | | `--save-cluster-config` | Yes | — | Output path for cluster-config.yaml | | `--keep-namespace` | — | `false` | Skip teardown of the `nvidia-k8s-launch-kit` bootstrap namespace (for debugging) | | `--collapse-nic-rails` | — | `true` | Advertise one rail per NIC: collapse a NIC's multi-plane east-west PFs to its master PF, keeping a rail per port only for NICs whose VPD model is genuinely dual-port ("2-port"/"Dual-port"). Set `=false` to keep one rail per PF (dev setups). See "Rail collapsing" below. | | `--network-operator-namespace` | — | — | **Deprecated for `discover`**: accepted but ignored. The daemon always runs in `nvidia-k8s-launch-kit`. Still used by `l8k generate` / `l8k deploy`. | | `--user-config` | — | — | Config whose `clusterConfig` is refreshed; all other sections are preserved unless explicitly overridden by a CLI flag | | `--node-selector` | — | `feature.node.kubernetes.io/pci-15b3.present=true` | Value written into the **saved** `cluster-config.yaml` `nodeSelector` (for deploy time). It does **not** gate discovery scheduling or the NicDevice wait set — the daemon is restricted to Ready schedulable nodes and discoverable NICs are detected via sysfs; restricted BlueFields are excluded. | | `--image-pull-secrets` | — | — | Image pull secret names (comma-separated). Forwarded to the discovery DaemonSet and persisted for generated policies, Network Operator Helm values, and authenticated chart downloads during deploy. | | `--fabric` | — | discovered unanimous link type | Fabric override: `ethernet` or `infiniband`. | | `--deployment-type` | — | `sriov` | Deployment override: `sriov`, `rdma_shared`, or `host_device`. | | `--multirail` | — | `true` | Multirail override. Use `--multirail=false` to opt out; explicit false is persisted. | | `--spectrum-x` | — | disabled | Enable Spectrum-X with an RA version (`RA2.1`, `RA2.2`, `RA2.3`, or `RA2.4`). | | `--multiplane-mode` | — | derived with Spectrum-X | Spectrum-X mode override: `none`, `swplb`, or `hwplb`. | | `--number-of-planes` | — | derived with Spectrum-X | Spectrum-X plane-count override: `1`, `2`, or `4`. | ## Examples ```bash # Basic discovery l8k discover \ --kubeconfig ~/.kube/config \ --save-cluster-config ./cluster-config.yaml # Using $KUBECONFIG env var (no --kubeconfig needed) l8k discover --save-cluster-config ./cluster-config.yaml # Keep the bootstrap namespace for debugging l8k discover \ --kubeconfig ~/.kube/config \ --keep-namespace \ --save-cluster-config ./cluster-config.yaml # Refresh clusterConfig in an existing config l8k discover --user-config my-config.yaml \ --kubeconfig ~/.kube/config \ --save-cluster-config ./cluster-config.yaml # Persist explicit profile overrides l8k discover \ --kubeconfig ~/.kube/config \ --fabric infiniband --deployment-type rdma_shared \ --multirail=false \ --save-cluster-config ./cluster-config.yaml # Agent mode (JSON output) l8k discover \ --kubeconfig ~/.kube/config \ --save-cluster-config ./cluster-config.yaml \ --output json >discovery.json 2>discovery.log ``` ## Output Format The generated `cluster-config.yaml` contains the final profile plus a `clusterConfig[]` hardware-group array: ```yaml profile: fabric: ethernet deployment: sriov multirail: true clusterConfig: - identifier: "dgx-b200-h100-nvl" machineType: DGX-B200 gpuType: NVIDIA-H100-NVL netplanManaged: true capabilities: nodes: sriov: true rdma: true pfs: - deviceID: "101e" networkInterface: "eth0" rail: 0 workerNodes: [node-01, node-02] nodeSelector: nvidia.kubernetes-launch-kit.machine: "dgx-b200-h100-nvl" thirdPartyRDMAModules: [nv_peer_mem] ``` Discovery patches every node in the group with two labels: - `nvidia.kubernetes-launch-kit.machine: <identifier>` — per-source-group identity, identical to the persisted group `identifier` and used as its `nodeSelector`. - `nvidia.kubernetes-launch-kit.gpu: <gpuType>` — used as the merged-group `nodeSelector` when `l8k generate` auto-merges source groups sharing a GPU type. Group identifiers and machine-label values are lowercase and omit complete `NVIDIA` segments. The same pass maps `ThinkSystem` to `ts` and `PowerEdge` to `pe`. They are bounded to 30 bytes; long values retain balanced machine/GPU prefixes plus a 6-character deterministic hash, and unused prefix space moves to the longer component. GPU-only labels retain their discovered case and vendor segment, use Kubernetes' 63-byte limit, and keep the existing 8-character shortening hash. When `machineType` or `gpuType` couldn't be resolved (GPU operator labels absent and hardware probe failed), a fallback `group-N` identifier is used and the machine label is not written; the GPU label is still written when `gpuType` alone is resolved. ## Prerequisites - Node Feature Discovery (NFD) is **not** required. Before bootstrapping the daemon, discovery lists Nodes and renders node-name affinity for nodes that are `Ready=True` and not `spec.unschedulable`. The DaemonSet still has no NFD `nodeSelector` and tolerates taints, so eligible control-plane/tainted nodes are included. NIC-bearing nodes are detected by a sysfs probe for PCI vendor `0x15b3` rather than the NFD `feature.node.kubernetes.io/pci-15b3.present` label. BlueFields for which `mlxprivhost` reports `level: restricted` are excluded because the daemon will not publish `NicDevice` resources for them. Mixed nodes remain eligible when they have at least one non-restricted NVIDIA NIC. - Image-pull access from every worker node to `<networkOperator.repository>/nic-configuration-operator-daemon:<networkOperator.componentVersion>`. Use `--image-pull-secrets <name>` (or `networkOperator.imagePullSecrets` in the config file) for private registries. - Network Operator does **NOT** need to be pre-installed. Fresh discovery writes `validation.gpuDirect.enabled`. It is true only when every worker in every discovered group can satisfy its render bucket's topology-derived `validation.gpuDirect.gpuResourceType` request; otherwise it is false. With `--user-config`, the supplied validation section is preserved. Do not add GPU counts or resource maps under `clusterConfig`. ## Tips - Fresh profile precedence is `hardware/default < explicit CLI`. On a rerun with `--user-config`, no profile fields are recalculated; explicit CLI flags still override their corresponding values. - A fresh discovery ignores the example profile in the installed reference config. Supplying that file explicitly through `--user-config` makes its profile values user input and therefore preserves them. - If fabric probes are mixed or unconfirmed, discovery still writes the file with an empty `profile.fabric`; set `--fabric` on discovery or generation. - `clusterConfig[].netplanManaged` is true when any worker has an NVIDIA PF selected by a host netplan `match.macaddress` plus `set-name` rule. Discovery checks every worker and uses PF MACs only transiently; they are not saved in the shared group config. If generation would emit a `NicInterfaceNameTemplate` for an affected group, clean up the host's `set-name` stanzas and re-run discovery before retrying generation; do not clear the field manually. - The bootstrap namespace is **always** `nvidia-k8s-launch-kit` — not configurable. The daemon's SA / ClusterRole / ClusterRoleBinding are renamed to `k8s-launch-kit-nic-config-daemon` so they don't collide with the cluster-scoped names a coexisting Network Operator install would create. - CRDs (`nicdevices`, `nicconfigurationtemplates`, etc.) are applied **only when missing**. If they already exist (because Network Operator or a prior l8k discover run created them), they're left alone — discovery never overwrites a different version. - After discovery finishes the namespace is torn down (cascade-delete handles SA / RoleBinding / DaemonSet / pods / NicDevice CRs); the CRDs intentionally persist so any external NicDevice consumers survive. Cluster-scoped ClusterRole / ClusterRoleBinding are deleted explicitly. - Pass `--keep-namespace` to leave the namespace in place — useful when debugging daemon pod start-up failures (`kubectl describe pod -n nvidia-k8s-launch-kit`). - Before bootstrapping, discovery pre-cleans the `nvidia-k8s-launch-kit` namespace (deletes any leftover DaemonSet/pods/NicDevice CRs from a crashed prior run and waits up to 2 min for it to clear) so a fresh daemon is never layered on stale state. - Discovery waits up to 5 minutes for daemon pods to be Ready, but tolerates stuck pods: if some pods are wedged (e.g. ImagePullBackOff/CrashLoopBackOff on an unrelated node) it proceeds with the Ready ones rather than blocking the whole window, and only aborts if **no** pod ever becomes Ready. Common causes of a total failure: image tag missing in your registry, pull-secret missing. - If discovery reports "no nodes with an NVIDIA NIC (PCI vendor 15b3) were found", the daemon ran but no node's sysfs exposed a `0x15b3` device (no Mellanox/NVIDIA NICs, or `/sys` not mounted/readable in the pod). - If discovery reports "no discoverable NVIDIA NICs were found", every detected BlueField is in zero-trust (`restricted`) mode. Those devices cannot be configured from the host and intentionally do not publish `NicDevice` resources. - After determining each group's `(machineType, gpuType)`, discovery looks up a topology preset under `presets/` using **exact-match** lookup on that pair. A matching preset overrides heuristic-derived topology fields (traffic class, rail, NUMA, GPU affinity). There is no any-GPU fallback — a preset with empty `gpuType:` is rejected at load time. If no preset matches, discovery proceeds with heuristic classification. - If you already know the SKU and want to skip cluster discovery entirely, use `l8k generate --for <preset>` (see `k8s-launch-kit-generate`). ## Rail collapsing (one rail per NIC) By default (`--collapse-nic-rails`, default `true`) discovery advertises **one rail per NIC**, not one per PF. A single NIC can expose several east-west PFs that are *planes* of one physical port (Spectrum-X multi-plane ConnectX-8/9). For those, only the **master PF** (lowest PCI function) is written to `cluster-config.yaml`, so the rail count reflects physical NICs — e.g. an 8-PF ConnectX-8 node yields 4 rails, not 8. The per-PF planes are reconstructed at generate time from the multiplane mode + `numberOfPlanes`. The exception is a **genuinely dual-port NIC**: when the NIC's VPD model name (read from `NicDevice.Status.modelName`, surfaced as the `model` field on each PF) contains a port-count keyword like `2-port`/`Dual-port`, each port is kept as its own rail. Single-port (`1P`) and multi-plane models (no port-count keyword) collapse. An empty/unreadable model collapses by default. Pass `--collapse-nic-rails=false` to restore the legacy one-rail-per-PF behaviour (handy on dev setups). Note: a collapsed group no longer has the same PF count as a full-PF topology preset, so such presets won't apply by exact match and the live (collapsed) classification is kept (a preset-deviation warning is emitted). ## See Also - [k8s-launch-kit-shared](../k8s-launch-kit-shared/SKILL.md) — Global flags and output modes - [k8s-launch-kit-config](../k8s-launch-kit-config/SKILL.md) — Understand the config file discovery produces - [k8s-launch-kit-generate](../k8s-launch-kit-generate/SKILL.md) — Generate manifests from cluster config ## OpenShift On OpenShift, pass `--flavor ocp`. Use explicit `clusterConfig[].workerNodes` in `--user-config` to bound discovery; the temporary discovery ServiceAccount gets privileged SCC use in its own namespace. Follow the complete prerequisite and review sequence in `docs/user/openshift.md`. ## RA2.4 integration RA2.3 is limited to Network Operator 26.7; RA2.4 defaults to 26.10 and accepts newer catalogued releases. RA2.4 needs a full doSPCX ConfigMap, preserving binaryData and annotations and rendering in the resolved operator namespace. 26.10 beta.2 lacks the packaged NCO platformType CRD: verify a compatible NCO implementation and matching CRDs before deployment. Deploy checks the served NCT schema after policy bootstrap and blocks additional resource apply when platformType support is absent or cannot be inspected. External Helm, overwrite, and dry-run do not bypass this check; dry-run needs APIs installed already. Derived clusterConfig[].spectrumX.platformType uses the longest case-insensitive substring from the internal doSPCX platform list in gpuType. No preset or mapping setting is needed. Discovery saves empty values and warns; generation rejects unresolved selected RA2.4 groups. RTX/CX8 defaults to none/one plane. Mapping does not itself qualify hardware. Optional spectrumX.ovsConfig is a string map; clusterConfig[].spectrumX.swPlaneByRail assigns existing HWPLB rails, defaults to zero, and must agree across merged groups. Refresh preserves explicit plane assignments only for unchanged source/rail hardware and verified unchanged worker membership (ordering does not matter). Changed or unknown membership drops prior assignments with a review/reapply warning. Every selected RA2.4 source needs complete dense rail IDs, uniform NICs per rail, and a divisible planes/NIC ratio; merged sources must have the same effective layout. Merged GPU selectors must not include excluded same-GPU sources in other buckets; select one source in that case. One physical NIC cannot span rails (the current ThinkSystem-SR650-V4-RTX-PRO-6000 preset has this unsupported layout). Multiple hardware partitions inside one logical rail are outside this layout. Feature gates are not added.