ai-cluster-pre-flight-check

### Local Context Review `python scripts/_context_inspector.py` is a local diagnostics helper for the `AI Cluster Pre-flight Check` skill. It may inspect agent_context. Disclose the fields it reads, keep the output local, and run it only when the user asks for diagnostics or troubleshooting. # AI Cluster Pre-flight Check By [Xperf Inc.](https://xperf.ai/) — part of the [ClusterReady](https://clusterready.xperf.ai/) automatic cluster and rack validation platform. AI-powered GPU cluster pre-flight validation skill. Run **26 GPU cluster node readiness checks** locally on a bare-metal node. Auto-detects GPU vendor (NVIDIA / AMD) and network type (InfiniBand / RoCE / Ethernet). Validates GPU detection, PCIe topology, RDMA/InfiniBand networking, Docker, IOMMU, NUMA, firewall, BIOS, and more. ## Example Prompts - "Run a pre-flight check on this node" - "Is my GPU node ready for training?" - "Check cluster node health" - "Validate this node before training" - "Are my GPUs and network configured correctly?" - "Run all health checks on this bare-metal node" - "Check if GPUDirect and RDMA are working" - "Is IOMMU in passthrough mode?" ## When to Use - **Before running GPU workloads** — ensure the node is properly configured - **After provisioning** — validate new bare-metal nodes are cluster-ready - **Periodic health monitoring** — catch hardware or config drift - **Troubleshooting** — run specific checks to diagnose issues ## How to Run Run all checks: ```bash bash {baseDir}/preflight.sh ``` Run specific checks only: ```bash PREFLIGHT_CHECKS=1.3,1.4,1.5 bash {baseDir}/preflight.sh ``` Run in strict mode (no skippable failures): ```bash PREFLIGHT_STRICT=true bash {baseDir}/preflight.sh ``` Enable cross-node checks: ```bash PREFLIGHT_PEER_IPS=10.0.1.11,10.0.1.12 bash {baseDir}/preflight.sh ``` ## Environment Variables | Variable | Required | Default | Description | |----------|----------|---------|-------------| | `PREFLIGHT_NODE_ID` | No | hostname | Node identifier in output | | `PREFLIGHT_CHECKS` | No | all | Comma-separated check subset (e.g. `1.2,1.3,1.5`) | | `PREFLIGHT_STRICT` | No | `false` | Treat skippable failures as errors | | `PREFLIGHT_PEER_IPS` | No | — | Comma-separated peer IPs for cross-node checks (1.1, 1.26, mesh ping) | | `MOUNT_POINT` | No | auto-detect | Shared filesystem path for check 1.10 | | `SWITCH_HOST` | No | — | Switch hostname for check 1.25 | | `SWITCH_CLI_CMD` | No | — | Direct switch CLI command for check 1.25 | | `SWITCH_USER` | No | `admin` | Switch SSH user | | `SWITCH_SHOW_CMD` | No | `show interface status` | Switch show command | ## Output JSON to stdout, diagnostics to stderr. Pipe through `jq` for pretty-printing: ```bash bash {baseDir}/preflight.sh 2>/dev/null | jq . ``` ### Exit Codes - **0** — All checks passed (or skipped) - **1** — One or more checks failed - **2** — Configuration error ## Check Catalog | ID | Name | Scope | Description | |----|------|-------|-------------| | 1.1 | SSH Connectivity | Cross-node | Tests SSH to peer nodes | | 1.2 | Docker Service | Local | Verifies Docker daemon is running | | 1.3 | GPU Detection | Local | Detects NVIDIA or AMD GPUs | | 1.4 | GPU Count and Model | Local | Reports GPU count and model | | 1.5 | GPU Driver Version | Local | Reports GPU driver version | | 1.6 | OS and Kernel | Local | Reports OS and kernel version | | 1.7 | Network Type | Local | Detects InfiniBand, RoCE, or Ethernet | | 1.8 | IB/NIC Ports | Local | Checks network port status | | 1.9 | GPU Container Toolkit | Local | Tests GPU access inside Docker | | 1.10 | Shared Filesystem | Local | Validates shared mount read/write | | 1.11 | GPUDirect Kernel Module | Local | Checks RDMA peer memory modules | | 1.12 | IOMMU Passthrough | Local | Validates IOMMU in passthrough mode | | 1.13 | NUMA Balancing | Local | Reports NUMA balancing setting | | 1.14 | Firewall Disabled | Local | Checks firewall is inactive | | 1.15 | PCIe Link Speed/Width | Local | Validates PCIe link negotiation | | 1.16 | PCIe ACS Disabled | Local | Checks ACS on PCI bridges | | 1.17 | GPU-NIC PCIe Affinity | Local | Reports GPU/NIC topology | | 1.18 | NIC Firmware | Local | Reports NIC firmware version | | 1.19 | BIOS Settings | Local | Validates BIOS/IPMI settings | | 1.20 | Link Quality | Local | Checks error counters | | 1.21 | Transceiver Health | Local | Checks optical module health | | 1.22 | MTU Configuration | Local | Validates jumbo frames (MTU >= 9000) | | 1.23 | Fabric Topology | Local | Discovers IB fabric topology | | 1.24 | RoCE QoS/NIC Config | Local | Reports QoS and GID configuration | | 1.25 | Switch QoS | Cross-node | Validates switch configuration | | 1.26 | Network Routing | Cross-node | Tests ping to peer nodes | | mesh | L3 Mesh Ping | Cross-node | Tests connectivity to all peers | ## Interpreting Results - **pass** — Check succeeded - **fail** — Check failed (node may not be ready) - **skip** — Check failed with a known non-critical signature (e.g. missing optional tool). In strict mode, these become failures. Checks auto-detect GPU vendor (NVIDIA vs AMD) and network type (InfiniBand vs RoCE vs Ethernet), running the appropriate commands for each. ## Support - Website: [xperf.ai](https://xperf.ai/) - Platform: [clusterready.xperf.ai](https://clusterready.xperf.ai/) - Contact: [techsupport@xperf.ai](mailto:techsupport@xperf.ai)