k8s-launch-kit-validate

# l8k: Validate > **PREREQUISITE:** Read `../k8s-launch-kit-shared/SKILL.md` for install paths, global flags, and exit codes. This workflow uses the default `host` target; `--target host` is equivalent. The CLI snapshots validation arguments and runs the standalone Host validation and report service through the target registry; checks and report semantics are unchanged. Verify that a previously generated and deployed NVIDIA networking deployment is correctly applied and matches the selected Network Operator release. Apply `docs/user/validation.md#worked-acceptance-decisions` to the report; compare intended and completed worker/rail coverage before acceptance. ## What it checks 1. **Network Operator Helm release version.** Reads the chart's `appVersion` from any release Secret named `sh.helm.release.v1.<release>.v<N>` whose release name contains "network-operator", in the operator namespace. Compares with the version expected by `networkOperator.selectedRelease` in `cluster-config.yaml` (looked up in l8k's embedded release catalog). 2. **Manifest state.** Every YAML manifest under `--deployment-files` (skipping example workloads and Helm `values.yaml`) is fetched from the cluster and classified by the per-Kind resource-state registry. `NicConfigurationTemplate` and `NicFirmwareTemplate` wait for the operator-populated `status.nicDevices` list to reflect current node, NIC type, PCI-address, serial-number, and part-number selectors and validate only the named devices. Their propagated template payload and condition `observedGeneration` must also be current; unrelated discovered NIC configuration state is ignored. A configuration template checks `FirmwareUpdateInProgress` only for a matched device with `spec.firmware`, so a stale firmware condition does not block a deployment with no `NicFirmwareTemplate`. Each manifest is reported `READY`, `IN-PROGRESS`, `ERROR`, or `MISSING`. 3. **Connectivity matrix.** By default, `l8k validate` applies the generated example DaemonSet, waits for ready pods, and runs source-bound `icmp`, `rping`, and `ib_write_bw` tests. When `validation.gpuDirect.enabled` is true and `ib_write_bw` is selected, a distinct DMA-BUF bandwidth family uses the connected GPU index for each source and destination rail. Profile templates declare both validation containers: the release-specific full-runtime DOCA image for the RDMA checks and `netshoot` for ICMP and route checks. GPUDirect-enabled templates set `LD_LIBRARY_PATH` on the DOCA container to prefer host-injected NVIDIA driver libraries over bundled CUDA compatibility libraries. Validate applies the generated DaemonSet without runtime container injection. The default mode is `strict`. If `--deployment-files` contains only user-provided `*example*.yaml` test DaemonSets, run connectivity only. If `values.yaml` or any non-example YAML is also present, run the complete validation pipeline. The DaemonSet does not need to be generated by Launch Kit and no separate manifest flag is used. Validation also reports deploy-preflight drift without remediating it. Validate constructs one strict desired-artifact snapshot before cluster checks. Malformed or nameless documents fail with source details and exit 2; `values.yml` and case variants must be renamed to `values.yaml`. `--wait`, connectivity, and reports reuse that desired snapshot while reading fresh cluster state. `SriovNetworkPoolConfig`, `SriovNetworkNodePolicy`, and `OVSNetwork` objects labeled with `spectrumx.nvidia.com/owner-name` are excluded from stray results because the Spectrum-X operator owns them as children of `SpectrumXRailPoolConfig`. Exit code is non-zero (4) on any missing manifest, version mismatch, or gating connectivity failure. A skipped, empty, or incomplete connectivity-only matrix also exits 4; every selected check family must produce a gating test, and non-gating cross-rail observations do not count. Version checks soft-skip when prerequisites are absent, but connectivity requires a user-owned cluster config. If `networkOperator.skipHelmChart: true` is set, or the user passes `--skip-network-operator-helm`, both Helm release/version and values checks are reported as skipped. Component-version, manifest, stray-resource, connectivity, and report stages still run. ## Usage ```bash l8k validate [--user-config <PATH>] [--deployment-files <DIR>] [--kubeconfig <PATH>] ``` ## Flags | Flag | Default | Description | |------|---------|-------------| | `--kubeconfig` | `$KUBECONFIG` | Path to kubeconfig with read access to the cluster | | `--user-config` | `./cluster-config.yaml` | User-owned cluster config; connectivity requires explicit `profile.routing` and `validation.gpuDirect.enabled` | | `--deployment-files` | `./deployment` | Directory containing generated manifests and `*example*.yaml` connectivity test DaemonSets; a test-only directory selects connectivity-only validation | | `--skip-network-operator-helm` | `networkOperator.skipHelmChart` (`false`) | Skip Helm release/version and values validation only | | `--validation-mode` | `validation.mode` (`strict`) | Connectivity mode: `quick`, `full`, or `strict` | | `--validation-checks` | `validation.checks` (`icmp,rping,ib_write_bw`) | Comma-separated connectivity checks; `""` disables all | | `--connectivity-timeout` | automatic (`0`) | Total connectivity budget is calculated from the generated matrix plan; set a positive duration for an explicit hard setup and execution deadline | | `--rdma-rping-iterations` | `validation.rdma.rpingIterations` | rping client iteration count | | `--rdma-ib-write-size` | `validation.rdma.ibWriteSize` | ib_write_bw message size | | `--rdma-ib-write-min-bandwidth-gbps` | `validation.rdma.ibWriteMinBandwidthGbps` | Minimum peak Gbps; `0` disables bandwidth gating | | `--junit-path` | disabled | Write JUnit XML alongside text/JSON; connectivity needs `profile.fabric: ethernet` or `infiniband` | | `--log-level` | disabled | `debug` for structured progress and timing; `trace` also includes bounded command output | ## Connectivity Modes - `quick`: all same-rail node pairs plus one non-gating cross-rail canary per source-rail/destination-rail mapping. - `full`: every source rail × every destination rail × every ordered pod pair; cross-rail results are reported but do not gate pass/fail. - `strict`: full matrix. Cross-rail gates by `profile.routing`: `source-based` must succeed, `destination-based` must stay isolated. All checks are source-bound. Before every ICMP probe, validate confirms that `ip route get <dst> from <src>` selects the requested source interface. A route using another interface is reported as not connected for that rail pair rather than forcing traffic onto it. When the source rail is selected, ICMP uses `ping -I <src-ip>` so source-based policy rules participate naturally. Routing diagnostics expect the source interface for source-based routing and the destination interface for destination-based routing. `rping` uses `-I <src-ip>`, and `ib_write_bw` uses `--bind_source_ip <src-ip>`. GPUDirect adds `--use_cuda=<endpoint-index> --use_cuda_dmabuf` independently on the client and server. Treat missing or ambiguous `connectedGPU` topology as a failure; never substitute GPU 0. Pull Secrets come from `networkOperator.imagePullSecrets` and must exist in every validation namespace. With the default `--connectivity-timeout=0`, validate logs one total automatic budget after planning the matrix. The calculation reflects the enabled test families, per-command limits, ordered pod-pair batches, bounded workload setup, cleanup, and a safety margin. A positive flag value replaces that calculation with a user-supplied hard deadline. The minimum connectivity config is: ```yaml profile: routing: source-based # or destination-based validation: gpuDirect: enabled: false ``` The test workload must be an `apps/v1` DaemonSet with a name, namespace, `netshoot` route helper, and an RDMA tools container when RDMA checks are selected. When GPUDirect is enabled with `ib_write_bw`, each `clusterConfig` group must list workers and east-west PFs with a non-negative `rail` and `connectedGPU: GPU<N>`. ## Examples ```bash # Defaults: ./cluster-config.yaml + ./deployment, $KUBECONFIG l8k validate # Explicit paths l8k validate --user-config ./cluster-config.yaml \ --deployment-files ./deployment \ --kubeconfig ~/.kube/config # Connectivity only from user-provided *example*.yaml DaemonSets l8k validate --user-config ./cluster-config.yaml \ --deployment-files ./connectivity-test \ --kubeconfig ~/.kube/config # Agent mode (JSON objects on stdout, logs on stderr; preserve l8k status) l8k validate --output json >validation.jsonl 2>validation.log && jq -s . validation.jsonl # Diagnose stage or batch progress without raw command output l8k validate --log-level debug # Capture bounded route, ICMP, RDMA client, and RDMA server evidence l8k validate --log-level trace --keep ``` Debug logs show the endpoint inventory, plan, source-route cache statistics, static checks, stages, RDMA batches, cleanup, report writes, elapsed time, and remaining timeout. Trace adds bounded commands and per-test stdout/stderr. Failed RDMA server logs are collected before the temporary files and test workload are removed. Add `--keep` only when follow-up pod inspection is needed. ## JUnit report Use `l8k validate --output json --junit-path ./reports/validation.xml` to retain both formats. The XML has a suite per catalog connectivity family, with individual directional pod/rail testcase children and suite counters. Disabled or unexecuted checks are skipped with reasons; existing gating semantics apply. Reports retain partial results and execution errors. A write failure fails an otherwise successful command; existing validation failure codes are preserved. Coverage failures remain visible alongside other failures, and preset cases include matching and skipped groups. See `docs/user/validation.md#junit-xml` for names, timing and interpretation. Preserve the process exit status. ## Output Text mode prints a short report: ``` Network Operator release selectedRelease: 26.4 expected version: v26.4.0-beta.6 deployed: network-operator (chart=26.4.0-beta.6 app=v26.4.0-beta.6 rev=3 status=deployed) result: MATCH Manifests [READY ] NicClusterPolicy/nic-cluster-policy in (cluster-scoped) [IN-PROGRESS] NicConfigurationTemplate/spectrum-x-config in network-operator — waiting for nic-configuration-operator to populate status.nicDevices with matched devices [MISSING ] SriovNetwork/sriov-network-rail-0 in default — not found in cluster ... Summary: 1/3 ready, 1 in-progress, 0 error, 1 missing; version: match; topology mismatches: 0 group(s) ``` JSON mode emits a sequence of objects. Full validation emits `versionCheck`, `manifests`, and `summary`; connectivity and the HTML `reportPath` are separate objects when those stages run. Parse with `jq -s` or a streaming parser and preserve the process exit status. `summary.success` is not the final verdict. Full Kubernetes validation can return success with skipped/incomplete connectivity; require explicit coverage evidence before accepting deployment. See `docs/user/validation.md#acceptance-outcomes`. Omit `--user-config` to reuse the bundle's `.l8k/resolved-config.yaml`. An explicit config overrides that metadata. Connectivity accepts the sidecar or a user-owned config with explicit routing and GPUDirect decisions. ## When this skill activates Trigger phrases include: "validate my deployment", "is my cluster correct", "are all the manifests applied", "does the chart version match", "did the deploy succeed", or any discrepancy claim about expected vs deployed state. ## See Also - [k8s-launch-kit-deploy](../k8s-launch-kit-deploy/SKILL.md) — apply manifests - [k8s-launch-kit-troubleshoot](../k8s-launch-kit-troubleshoot/SKILL.md) — investigate failures uncovered by validate - [k8s-launch-kit-shared](../k8s-launch-kit-shared/SKILL.md) — global flags and exit codes ## OpenShift OpenShift validation checks Network Operator CSV version and creates a temporary `l8k-validation-*` namespace for each source workload namespace. It copies required NADs and image pull secrets and creates dedicated ServiceAccount/SCC/RBAC there. The kubeconfig needs namespace/SCC creation and source NAD/Secret read permissions. Cleanup removes the temporary resources; `--keep` retains them. Inspect exact retained names after interrupted cleanup. OpenShift with connectivity enabled requires at least one gating test per selected family; in-progress manifests and incomplete coverage fail. Other profiles beyond SR-IOV Ethernet still need live hardware qualification; see `docs/user/openshift.md`. ## Optional standard NIC tuning `nicConfigurationOperator.deployNicConfigurationTemplate` defaults to false. Root, generate and discover accept `--deploy-nic-configuration-template[=false]`. It enables NCO independently of interface naming and renders per-source, east-west PCI-scoped templates for standard SR-IOV, RDMA-shared and host-device profiles on k8s/OCP. VF count comes from `sriov.numVfs`; link type comes from `profile.fabric`. Ethernet gets DSCP trust and priority-3 PFC; InfiniBand omits RoCE tuning. Spectrum-X rejects this option. Only SR-IOV disables the Mellanox plugin (Helm values on k8s, additive `disablePlugins` merge on OCP). Kubernetes external Helm owners must configure that plugin themselves. Deploy and validate consume the generated template and require all matched NicDevices to report current `ConfigUpdateInProgress=False` with reason `UpdateSuccessful`; stale/partial status waits and failures surface. Opting out does not restore applied NIC settings. On k8s, omitted templates (also for unselected groups) block deploy preflight unless `--overwrite-existing` is used, which deletes them. OCP preserves existing templates and disabled plugins. Generation rejects inventory with selected and excluded ports on the same PCI device. Review whole-NIC scope, including ports absent from inventory, and qualify runtime RoCE support for the installed NCO/DOCA/RHCOS versions before live use. See `docs/reference/configuration.md#nic-configuration-and-naming`.