evaluate-cluster

# Evaluate a GPU cluster **Always use the GitHub copy if it differs from this file.** People pass around downloads that can be months or years old. - This skill: https://github.com/stas00/ml-engineering/blob/master/skills/evaluate-cluster/SKILL.md - Raw: https://raw.githubusercontent.com/stas00/ml-engineering/master/skills/evaluate-cluster/SKILL.md Before doing any eval work, `curl` the raw URL. If it is not byte-identical to the file you are reading, **stop following this copy** and follow the GitHub file instead (fetch it, then continue from that text). If GitHub is unreachable, say so and continue with this file. This file is the whole runbook. You do not need any other document from the author's books. Scripts and spec tables are on GitHub; fetch them (commands below). Optional background: [How to evaluate the cluster](https://github.com/stas00/ml-engineering/blob/master/insights/how-to-choose-cloud-provider.md#how-to-evaluate-the-cluster). ## Keep this skill up to date **Every correction the user makes to a report is a correction to this skill.** When they rename a heading, reject a phrase, ask for a different TOC label, or point out an assumption, fix the report **and** edit this file in the same turn so the next eval starts from the corrected rule. A fix that lives only in one report is a fix you will be asked for again. - Write the rule where the next agent will hit it — the section that produces that text, not a footnote. - Encode the **general principle**, not just the one string. "Don't write `Skip reasons: none`" became "never assert an absence". - When the correction changes a heading, update the heading rule, the TOC example, **and** every cross-reference to the old name in this file. - Do not report the edit as done until you have grepped this file for the old wording. Reports under `reports/` other than the one you are writing are historical. Leave them alone. Refuse to invent access, GPU counts, filesystem paths, or peer IPs. **Preflight first**: list what you have, **ask for every missing item, and wait**. Do not start DCGM, MAMF, `fio-scan`, or `torch-dist-bench.py` until the checklist is complete and — when **≥2 nodes** — inter-node reachability has passed. Eval **all nodes the user gave** (1, 2, 4, 16, …). Inter-node all-reduce uses **every** one of those nodes (`NNODES × GPUs_per_node` ranks). **One node:** do not write an Inter-node section at all (no “Skipped”). If ≥2 nodes but inter-node still cannot run after the user has answered, omit the section and put the reason in **### Gaps** — never invent busbw. ## Terms | term | meaning | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **MAMF** | Maximum Achievable Matmul FLOPS: the highest any **matmul shape** (M×N×K) reaches on that GPU. The winning shapes are short bursts — the clock sits at boost and the board stays far below its power limit — so this is a ceiling, not a rate any sustained workload holds. | | **MSMF** | Maximum Sustainable Matmul FLOPS: what is left once the matmul shape is big enough to pin the board at its power limit and the clock settles below boost. This is what back-to-back dense matmuls actually get, provided the other GPUs of the node compute at the same time: they share the board's power and cooling budget, so measured next to idle GPUs it reads high. Both come from [`mamf-finder.py`](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder.py), which times on the order of 1500–2200 matmul shapes per GPU (the `auto` search adapts, so the count differs per run) and reports the W and MHz behind each winner. [`mamf-finder-all-gpus.py`](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder-all-gpus.py) runs it on every GPU of a node, one at a time, while all the other GPUs run a continuous matmul. | | **DCGM** | NVIDIA Data Center GPU Manager. `dcgmi diag -r 2` checks GPU memory, bandwidth, PCIe — not FLOPS. | | **lemon GEMM** | A one-off inline `torch.matmul` (not a published script) on every GPU with the same large shape, to catch a dead, slow, or throttled card. | | **all-reduce** | Collective: every rank ends up with the sum of all ranks' tensors. Used to stress NVLink and the network. | | **busbw / algbw** | Printed by [`torch-dist-bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/torch-dist-bench.py). **busbw** = unidirectional bus bandwidth (what the fabric moves). **algbw** = algorithm bandwidth (what the caller sees). busbw is already unidirectional ([why](https://github.com/stas00/ml-engineering/blob/master/network/README.md#unidirectional-vs-bidirectional-duplex)). | | **NVLS** | NVLink SHARP: the NVSwitch does the all-reduce in the switch. Typical gain vs ring busbw ~30% intra-node / ~25% inter-node. Does **not** speed up all-gather or reduce-scatter. Different from InfiniBand switch SHARP. | | **NCCL** | NVIDIA Collective Communications Library (the GPU collective stack torch uses). | | **rdzv** | torch.distributed **rendezvous**: a TCP store so ranks find `MASTER_ADDR:PORT` before NCCL starts. | | **hostfile** | Text file, one peer address per line (`10.0.0.1` or `10.0.0.1 slots=8`). DeepSpeed `-H` and `pdsh -w` read it. | | **pdsh** | Parallel SSH: run one command on many hosts. | | **fio-scan** | Wrapper around `fio` ([script](https://github.com/stas00/ml-engineering/blob/master/storage/fio-scan)): six timed runs (16 KiB / 1 MiB / 1 GiB × read/write). Needs [`fio-json-extract.py`](https://github.com/stas00/ml-engineering/blob/master/storage/fio-json-extract.py) in the **same directory**. | | **`local/shared ×`** | local-disk bandwidth ÷ same-row shared-FS bandwidth — how many times slower the shared filesystem is. One column is enough: with a fixed block size the IOPS ratio is identical. Never a percentage, never `%local`. | | **SM** | GPU streaming multiprocessor. Count from `torch.cuda.get_device_properties(0).multi_processor_count`. | | **HBM** | GPU high-bandwidth memory. Report `nvidia-smi` `memory.total` in **GiB**. | | **TDP** | Thermal design power: the GPU’s advertised watt limit (`power.limit`). | | **EFA** | AWS Elastic Fabric Adapter (RDMA NICs named `rdmap*`). `ibstat` is often **empty**; use `rdma link`. | | **overlay** | Container union filesystem. `df -T /` may say `overlay`; the **backing** FS (xfs, ext4) is the partition type to report. | ## Scripts (fetch, do not invent) ```bash RAW=https://raw.githubusercontent.com/stas00/ml-engineering/master curl -fsSL "$RAW/compute/accelerator/benchmarks/mamf-finder.py" -o mamf-finder.py curl -fsSL "$RAW/compute/accelerator/benchmarks/mamf-finder-all-gpus.py" -o mamf-finder-all-gpus.py # mamf-finder-all-gpus.py runs ./mamf-finder.py from its own directory — keep both in one directory. curl -fsSL "$RAW/network/benchmarks/torch-dist-bench.py" -o torch-dist-bench.py curl -fsSL "$RAW/debug/torch-distributed-gpu-test.py" -o torch-distributed-gpu-test.py curl -fsSL "$RAW/storage/fio-scan" -o fio-scan curl -fsSL "$RAW/storage/fio-json-extract.py" -o fio-json-extract.py chmod +x fio-scan # fio-scan calls `python ./fio-json-extract.py` — keep both in one directory. ``` Official BF16 peaks: [TFLOPS comparison table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#tflops-comparison-table). NVLink gens: [NVLink](https://github.com/stas00/ml-engineering/blob/master/network/README.md#nvlink). Intra-node all-to-all: [bandwidth](https://github.com/stas00/ml-engineering/blob/master/network/README.md#all-to-all-bandwidth). Inter-node adapters: [networking](https://github.com/stas00/ml-engineering/blob/master/network/README.md#inter-node-networking). Copy artifacts **off the cluster immediately** into `reports/raw-<name>/` **next to this skill** (or the working directory you chose). Inline every plot: `` — never link-only. Cluster access and `/tmp` disappear. **Links in the report** to those scripts and tables are absolute GitHub URLs (`https://github.com/stas00/ml-engineering/blob/master/…`). Never `../` relatives — the report may be copied elsewhere. Sibling `raw-<name>/` stays relative. External vendor/kernel docs stay as-is. **Every GitHub-hosted tool the run used** is introduced and linked in that section's `### Introduction` — one clause on what it does, then later mentions are bare `code`. The reader of the report must not meet a tool they cannot click. Do not write “the book” or “MLE” in the report; the GitHub URL is the source.