菜单
Diagnose and fix NVIDIA Network Operator issues in Kubernetes. Use when pods can't get RDMA network, SR-IOV VFs not showing, IP allocation failures, network attachment missing, must-gather diagnostics, operator pods crashing, NicClusterPolicy errors, or IPAM pool exhaustion.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Configure netop-tools for a specific hardware platform and use case. Use when setting up global_ops_user.cfg, choosing a use case (sriovnet_rdma, sriovibnet_rdma, hostdev, ipoib, macvlan), selecting IPAM type, configuring NETOP_NETLIST device list, setting feature flags, or switching between platforms (DGX, Dell, Lenovo, OCI, SuperMicro).
Deploy NVIDIA Network Operator to a Kubernetes cluster via netop-tools. Use when installing network operator, setting up RDMA networking, deploying SR-IOV, running ins-network-operator.sh, generating Helm values, applying network CRDs, or troubleshooting deployment failures.
Manage SR-IOV virtual functions, PCI devices, link speed, and RDMA testing for NVIDIA Network Operator. Use when setting VFs, querying PCI devices, forcing link speed, running RDMA bandwidth tests, checking RDMA capability, managing node labels and taints, or cordoning nodes.
Run and create tests for netop-tools configuration generation. Use when running unitest.sh, validating YAML generation, creating new test cases, checking test baselines, debugging test failures, or verifying config changes don't break existing tests.
Upgrade NVIDIA Network Operator to a new version. Use when changing NETOP_VERSION, running upgrade-network-operator.sh, migrating between operator versions, handling CRD schema changes, or rolling back a failed upgrade.
| name | troubleshooting-network-operator |
| description | Diagnose and fix NVIDIA Network Operator issues in Kubernetes. Use when pods can't get RDMA network, SR-IOV VFs not showing, IP allocation failures, network attachment missing, must-gather diagnostics, operator pods crashing, NicClusterPolicy errors, or IPAM pool exhaustion. |
Systematic diagnostic flow for Network Operator issues.
# Full must-gather (saves to /tmp/nvidia-network-operator_YYYYMMDD_HHMM/)
./must-gather-network.sh
# Python CLI alternative
python3 python_tools/netop_tools.py must-gather --output-dir /tmp/diag
kubectl get pods -n ${NETOP_NAMESPACE}
./ops/getNicClusterPolicy.sh
./ops/getnetwork.sh
What to look for: All operator/daemon pods should be Running. NicClusterPolicy should show state: ready.
./ops/get-network-attach-defs.sh # Verify NetworkAttachmentDefinitions exist
./ops/getnetworkstatus.sh # Check pod network-status annotations
./ops/getpodnetworkstatus.sh # Pod-level details
./ops/checksriovstate.sh # Check sync status
./ops/syncsriov.sh # Wait for sync (up to 10 min)
./ops/getsriovnodepolicy.sh # Verify node policies
./ops/getsriovnodestate.sh # Per-node SR-IOV state
./ops/getnumvfs.sh # Query actual VF count on devices
./ops/checkipam.sh # Node IPAM annotations
./ops/checkippool.sh <NODENAME> # Per-node IP usage
./ops/getippool.sh # Pool definitions
./ops/getallocatedip.sh # Currently allocated IPs
# Python CLI
python3 python_tools/netop_tools.py ops check ipam
./ops/pingtest.sh # Pod-to-pod connectivity
./ops/check-iptables.sh # Firewall rules
./ops/chkfw.sh # Firewall status
| Symptom | Check | Likely Cause |
|---|---|---|
| Operator pod CrashLoopBackOff | kubectl logs -n ${NETOP_NAMESPACE} <pod> | Version mismatch, missing CRDs |
| No VFs on node | checksriovstate.sh | Sync in progress, wrong PCI BDF in NETOP_NETLIST |
| Pod stuck Pending | kubectl describe pod <pod> | Resource not available (wrong resource name) |
| IP not allocated | checkipam.sh | Pool exhausted, wrong IPAM_TYPE, pool not applied |
| Network annotation missing | getnetworkstatus.sh | NetworkAttachmentDefinition not in pod namespace |
| Finalizer blocking delete | getfinalizers.sh | Orphaned finalizer — patch to remove |
| MTU errors | Check NETOP_MTU in config | Mismatch between config and physical network |
./ops/getfinalizers.sh # Find stuck finalizers
./uninstall/delstucknamespace.sh # Force-delete terminating namespaces
./uninstall/delevictedpods.sh # Remove evicted pods
| File | Contents |
|---|---|
network_operator_pod.log | Operator controller logs |
network_operand_pod_*.log | Per-daemon logs (OFED, device plugin, IPAM) |
custom_resource_*.yaml | All CRD state snapshots |
nodes.yaml | Node labels, taints, allocatable resources |