# srt-slurm-recipes
This repository is the source of truth for official `srt-slurm` sweep configurations across node counts, models, GPU platforms, and inference frameworks.
The upstream tool is [NVIDIA/srt-slurm](https://github.com/NVIDIA/srt-slurm). Keep benchmark execution logic in that project; keep reusable sweep configuration in this repository.
## Layout
Recipes are organized by benchmark shape:
```text
recipes/
single-node/
<model>/
<gpu>/
<framework>/
multi-node/
<model>/
<gpu>/
<framework>/
```
Each leaf directory is reserved for one or more `*.yaml` sweep configs for that exact combination.
Current recipes:
Run `python scripts/update_recipe_table.py` after adding or removing recipe directories,
and `python scripts/update_agentx_table.py` for `agentic/` directories.
The single-node AgentX recipes are generated from the published InferenceX
results with `python scripts/generate_agentx_single_node.py --inferencex <path>`;
edit that script rather than the files it writes, and add `--refresh-points` to
re-query the dashboard API for newly published datapoints.
The same checks are available as a pre-commit hook at `.githooks/pre-commit`.
Enable it for a checkout with `scripts/install_hooks.sh`.
## AgentX
Ported SemiAnalysis InferenceX agentic-coding (`cc-traces-weka`) recipes.
- **Multi-node:** copies of the published InferenceX YAMLs. GB200 / GB300 NVL72
jobs that the dashboard labels **Single-node aggregate** (one node, a few
chips, aggregated serving) still live here, because InferenceX published them
as YAML under `benchmarks/multi_node/` rather than as bash launchers.
- **Single-node:** no published YAML. The published datapoints come from the
InferenceX dashboard API. This tree is B200 / B300 / H100 / H200 only.
### Requires an InferenceX checkout
These recipes carry the engine configuration only. The client side stays in
InferenceX: `benchmark.command` runs
[`agentic_srt.sh`](https://github.com/SemiAnalysisAI/InferenceX/blob/main/benchmarks/multi_node/agentic_srt.sh)
out of `/infmax-workspace`, which `srtctl` mounts from `INFMAX_WORKSPACE`.
Neither `srt-slurm` nor this repository vendors that stack, so AgentX recipes do
not run without it:
```bash
git clone --recurse-submodules https://github.com/SemiAnalysisAI/InferenceX.git
export INFMAX_WORKSPACE=/shared/path/to/InferenceX
```
`--recurse-submodules` is not optional. `agentic_srt.sh` sources
`benchmarks/benchmark_lib.sh`, which installs the `utils/agentic-benchmark`
requirements and `-e utils/aiperf`, and `utils/aiperf` is a submodule.
`INFMAX_WORKSPACE` is a host path bind-mounted into the container, so it must
resolve on the compute nodes, not just on the login node.
Then dry-run before submitting: it prints the resolved mount table, and flags a
missing `INFMAX_WORKSPACE` rather than letting the job reach exit 127 after the
workers have loaded.
```bash
srtctl dry-run -f <recipe>
srtctl apply -f <recipe> # every variant in the file
srtctl apply -f <recipe>:zip_override_conc[0] # one published concurrency
```
Legend:
- ✅ Recipes are present (links to the model or platform directory), followed by the frameworks covered and their recipe counts
- ➖ Not ported for that platform
Per-recipe index with InferenceX source links:
[multi-node](recipes/multi-node/AGENTX.md),
[single-node](recipes/single-node/AGENTX.md).
<!-- agentx-table:start -->
<!-- This section is generated by `python scripts/update_agentx_table.py`. -->
### Multi-Node
<table>
<thead>
<tr>
<th>Model</th>
<th><code>GB200 / GB300</code></th>
<th><code>B200 / B300</code></th>
<th><code>H100 / H200</code></th>
<th>Total</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>DeepSeek-V4-Pro</code></td>
<td><a href="recipes/multi-node/DeepSeek-V4-Pro/">✅</a> vllm 15, sglang 6, trtllm 6</td>
<td><span title="No AgentX recipe ported for this platform">➖</span></td>
<td><a href="recipes/multi-node/DeepSeek-V4-Pro/H200/">✅</a> sglang 1</td>
<td>28</td>
</tr>
<tr>
<td><code>Kimi-K3</code></td>
<td><a href="recipes/multi-node/Kimi-K3/">✅</a> vllm 16</td>
<td><a href="recipes/multi-node/Kimi-K3/B200/">✅</a> vllm 10</td>
<td><a href="recipes/multi-node/Kimi-K3/H200/">✅</a> vllm 3</td>
<td>29</td>
</tr>
<tr>
<td><code>MiniMax-M3</code></td>
<td><a href="recipes/multi-node/MiniMax-M3/">✅</a> vllm 16</td>
<td><span title="No AgentX recipe ported for this platform">➖</span></td>
<td><span title="No AgentX recipe ported for this platform">➖</span></td>
<td>16</td>
</tr>
<tr>
<td><code>Qwen3.5</code></td>
<td><a href="recipes/multi-node/Qwen3.5/">✅</a> sglang 18, trtllm 6</td>
<td><span title="No AgentX recipe ported for this platform">➖</span></td>
<td><span title="No AgentX recipe ported for this platform">➖</span></td>
<td>24</td>
</tr>
</tbody>
</table>
### Single-Node
<table>
<thead>
<tr>
<th>Model</th>
<th><code>B200 / B300</code></th>
<th><code>H100 / H200</code></th>
<th>Total</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>DeepSeek-V4-Pro</code></td>
<td><a href="recipes/single-node/DeepSeek-V4-Pro/">✅</a> vllm 8, sglang 5</td>
<td><span title="No AgentX recipe ported for this platform">➖</span></td>
<td>13</td>
</tr>
<tr>
<td><code>Kimi-K3</code></td>
<td><a href="recipes/single-node/Kimi-K3/B300/">✅</a> vllm 4</td>
<td><span title="No AgentX recipe ported for this platform">➖</span></td>
<td>4</td>
</tr>
<tr>
<td><code>MiniMax-M3</code></td>
<td><a href="recipes/single-node/MiniMax-M3/">✅</a> vllm 7, trtllm 5</td>
<td><a href="recipes/single-node/MiniMax-M3/">✅</a> vllm 4</td>
<td>16</td>
</tr>
<tr>
<td><code>Qwen3.5</code></td>
<td><a href="recipes/single-node/Qwen3.5/">✅</a> sglang 14</td>
<td><a href="recipes/single-node/Qwen3.5/">✅</a> sglang 4</td>
<td>18</td>
</tr>
</tbody>
</table>
<!-- agentx-table:end -->
## 1k/1k, 8k/1k Recipes
Fixed-ISL/OSL sweep recipes (`1k1k`, `8k1k`, and related ISL/OSL configs).
Recipes under an `agentic/` directory are not counted here — they are indexed in
[AgentX](#agentx) instead. A model whose only coverage is agentic therefore has
no row in this table: InferenceX sweeps Kimi-K3 on the agentic-coding workload
only, so it appears under AgentX and not below.
Legend:
- ✅ Recipe is present (links to containing directory)
- ➖ Not available (hover the dash for why: not offered on that platform, or does not fit a single node)
<!-- recipe-table:start -->
<!-- This section is generated by `python scripts/update_recipe_table.py`. -->
### Multi-Node
<table>
<thead>
<tr>
<th>Model</th>
<th><code>GB200 / GB300</code></th>
<th><code>B200 / B300</code></th>
<th><code>H100 / H200</code></th>
</tr>
</thead>
<tbody>
<tr>
<td><code>DeepSeek-R1</code></td>
<td><a href="recipes/multi-node/DeepSeek-R1/">✅</a></td>
<td><a href="recipes/multi-node/DeepSeek-R1/B300/">✅</a></td>
<td><a href="recipes/multi-node/DeepSeek-R1/">✅</a></td>
</tr>
<tr>
<td><code>DeepSeek-V4-Pro</code></td>
<td><a href="recipes/multi-node/DeepSeek-V4-Pro/">✅</a></td>
<td><a href="recipes/multi-node/DeepSeek-V4-Pro/">✅</a></td>
<td><a href="recipes/multi-node/DeepSeek-V4-Pro/H100/">✅</a></td>
</tr>
<tr>
<td><code>GLM-5</code></td>
<td><a href="recipes/multi-node/GLM-5/GB300/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>gpt-oss-120b</code></td>
<td><a href="recipes/multi-node/gpt-oss-120b/GB200/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>Kimi-K2.5</code></td>
<td><a href="recipes/multi-node/Kimi-K2.5/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>MiniMax-M2.5</code></td>
<td><a href="recipes/multi-node/MiniMax-M2.5/GB200/">✅</a></td>
<td><a href="recipes/multi-node/MiniMax-M2.5/B300/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>MiniMax-M3</code></td>
<td><a href="recipes/multi-node/MiniMax-M3/">✅</a></td>
<td><a href="recipes/multi-node/MiniMax-M3/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>NVIDIA-Nemotron-3-Ultra</code></td>
<td><a href="recipes/multi-node/NVIDIA-Nemotron-3-Ultra/GB200/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>Qwen3.5</code></td>
<td><a href="recipes/multi-node/Qwen3.5/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>Qwen3.8</code></td>
<td><a href="recipes/multi-node/Qwen3.8/GB300/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
</tbody>
</table>
### Single-Node
<table>
<thead>
<tr>
<th>Model</th>
<th><code>B200 / B300</code></th>
<th><code>H100 / H200</code></th>
<th><code>RTX-PRO-6000</code></th>
</tr>
</thead>
<tbody>
<tr>
<td><code>DeepSeek-R1</code></td>
<td><a href="recipes/single-node/DeepSeek-R1/">✅</a></td>
<td><a href="recipes/single-node/DeepSeek-R1/H200/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>DeepSeek-V4-Pro</code></td>
<td><a href="recipes/single-node/DeepSeek-V4-Pro/">✅</a></td>
<td><a href="recipes/single-node/DeepSeek-V4-Pro/H200/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>gpt-oss-120b</code></td>
<td><a href="recipes/single-node/gpt-oss-120b/B200/">✅</a></td>
<td><a href="recipes/single-node/gpt-oss-120b/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>Kimi-K2.5</code></td>
<td><a href="recipes/single-node/Kimi-K2.5/">✅</a></td>
<td><a href="recipes/single-node/Kimi-K2.5/H200/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>MiniMax-M2.5</code></td>
<td><a href="recipes/single-node/MiniMax-M2.5/">✅</a></td>
<td><a href="recipes/single-node/MiniMax-M2.5/H100/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>MiniMax-M3</code></td>
<td><a href="recipes/single-node/MiniMax-M3/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>NVIDIA-Nemotron-3-Super-120B-A12B</code></td>
<td><a href="recipes/single-node/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4/B200/">✅</a> NVFP4</td>
<td><a href="recipes/single-node/NVIDIA-Nemotron-3-Super-120B-A12B-FP8/H100/">✅</a> FP8</td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>NVIDIA-Nemotron-3-Ultra</code></td>
<td><a href="recipes/single-node/NVIDIA-Nemotron-3-Ultra/">✅</a></td>
<td><a href="recipes/single-node/NVIDIA-Nemotron-3-Ultra/H200/">✅</a></td>
<td><span title="Not available on this platform">➖</span></td>
</tr>
<tr>
<td><code>Qwen3.5</code></td>
<td><a href="recipes/single-node/Qwen3.5/">✅</a></td>
<td><a href="recipes/single-node/Qwen3.5/">✅</a></td>
<td><a href="recipes/single-node/Qwen3.5/RTX-PRO-6000/">✅</a></td>
</tr>
</tbody>
</table>
<!-- recipe-table:end -->
## Running a Sweep
Install `srt-slurm` from upstream, then point `srtctl` at a recipe YAML in this repo:
```bash
git clone https://github.com/NVIDIA/srt-slurm.git
cd srt-slurm
uv pip install -e .
RECIPES_PATH=/path/to/srt-slurm-recipes
FRAMEWORK=<framework>
uv run srtctl dry-run -f "${RECIPES_PATH}/recipes/single-node/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4/B200/${FRAMEWORK}/1k1k.yaml"
uv run srtctl apply -f "${RECIPES_PATH}/recipes/single-node/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4/B200/${FRAMEWORK}/1k1k.yaml" \
--tags official,NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4,B200,"${FRAMEWORK}"
```
The `dry-run` command should be used before submitting jobs.
## Contributing
See [CONTRIBUTING.md](.github/CONTRIBUTING.md) for how to add or update official sweep recipes.
## Support, Security & Governance
- **Support level:** Maintained - see [SUPPORT.md](.github/SUPPORT.md) for how to get help.
- **Security:** Report vulnerabilities privately per [SECURITY.md](SECURITY.md). Do not file public issues for security reports.
- **Code of Conduct:** [CODE_OF_CONDUCT.md](.github/CODE_OF_CONDUCT.md)
- **Governance & Maintainers:** [GOVERNANCE.md](.github/GOVERNANCE.md) · [MAINTAINERS.md](.github/MAINTAINERS.md)
- **Citation:** [CITATION.md](.github/CITATION.md)
## License
This repository follows the upstream `srt-slurm` license. See [LICENSE](LICENSE).