| name | pytorch-runners-routing |
| description | How PyTorch CI runners and jobs are defined and resolved across .github/workflows/ — test-matrix runner labels, the runner determinator (lf / arc / mt- prefixes and the PR #186663 lf+arc combination), the test-infra #5132 experiment config, EC2 -> ARC label translation via .github/arc.yaml and .github/scripts/map_ec2_to_arc.py (including the meta_only_runners override for H100/B200), build vs test runner decoupling, the _runner-determinator.yml / _linux-build.yml / _linux-test.yml flow, and per-workflow opt-in / opt-out mechanisms. Applies to ~/meta/pytorch. Load this skill whenever working on, researching, debugging, reviewing, or answering ANY question about PyTorch CI runners, runner labels, test matrices, runner prefixes (lf, mt-, lf-, amd-do-), experiments (lf / arc / amd-do), the runner determinator, ARC fleet routing, H100/B200/A100 runner placement, GitHub Actions runs-on values in .github/workflows/, or how jobs get assigned to specific runner pools.
|
PyTorch CI Runner Routing & Resolution
Scope
Everything about how .github/workflows/*.yml jobs in pytorch/pytorch land on a
specific runner — label definition, prefix resolution, fleet routing, the experiment
system, and the EC2-to-ARC translation pipeline. Read this BEFORE touching any
runs-on: value, any test-matrix: entry, any runner_prefix: interpolation, any
check_experiments: / opt_out_experiments: input, arc.yaml, or
runner_determinator.py / map_ec2_to_arc.py.
Mental Model: Four Inputs, One Final Label
Every PyTorch CI job runs on a runs-on: value that GitHub uses to pick a runner.
That label is assembled at runtime from four things:
- Bare label in the workflow YAML —
linux.c7i.2xlarge, linux.aws.h100, etc.
runner_prefix: input to the reusable workflow — interpolated from the
determinator's label-type output ("", "lf.", "mt-", "lf-", optionally
plus "amd-do-").use-arc: flag — triggers the EC2 -> ARC label translation step.arc.yaml mapping — the EC2 -> ARC translation table, plus the
meta_only_runners override list.
A label like linux.aws.h100 can become any of:
linux.aws.h100 (default fleet, no experiments)lf.linux.aws.h100 (LF experiment on, no ARC)mt-l-x86iamx-22-225-h100 (ARC experiment on — translated via arc.yaml)lf-l-x86iamx-22-225-h100 (ARC + LF both on, post PR #186663)
H100/B200 are special-cased: regardless of the four inputs above, they are pinned
to mt-<arc-label> via the meta_only_runners override (see Meta-Only Override
for H100/B200 below).
The 5 Files That Define Runner Routing
| File | Purpose |
|---|
.github/workflows/_runner-determinator.yml | Reusable workflow. Caller passes check_experiments / opt_out_experiments. Emits label-type (the prefix), use-arc, amd-do-label-type, runner-config, runner-type, runner-label, ci-docker-hash. |
.github/scripts/runner_determinator.py | The script. Fetches the rollout config from pytorch/test-infra#5132 (first comment), evaluates per-user opt-in/out + per-workflow allowlist + rollout %, and emits the prefix. |
.github/scripts/test_runner_determinator.py | Tests for the determinator. Run on PR changes to the script. |
.github/arc.yaml | EC2 -> ARC label mapping (runner_mapping) AND meta_only_runners override list for H100/B200. |
.github/scripts/map_ec2_to_arc.py + test_map_ec2_to_arc.py | Script that rewrites a test-matrix's runner: field from EC2 labels to ARC labels using arc.yaml. Invoked from _linux-build.yml. |
The Two Reusable Workflows That Consume the Determinator Output
| File | Role |
|---|
.github/workflows/_linux-build.yml | Build job. runs-on: ${{ inputs.runner_prefix }}${{ inputs.runner }}. Also runs map_ec2_to_arc.py --prefix ${{ inputs.runner_prefix }} over the test-matrix and re-emits it as the build job's test-matrix output. The translated matrix is what the test job consumes. |
.github/workflows/_linux-test.yml | Test job. Has two job branches gated on inputs.use-arc: test (EC2 path, EBS volumes) and test-osdc (ARC path, IAM role arn:aws:iam::308535385114:role/arc, container --gpus all). Both use runs-on: ${{ matrix.runner }} from the translated test-matrix. |
There are sibling workflows for other platforms (_mac-build.yml, _mac-test.yml,
_win-build.yml, _win-test.yml, _rocm-test.yml, _xpu-test.yml, _vllm-build.yml,
_vllm-benchmark.yml, _linux-test-stable-fa3.yml). Not all of them participate in
the determinator/ARC flow — Linux build + test are the primary path.
Caller Pattern (How a Workflow Plugs Into the Determinator)
Every workflow that wants experiment-driven runner placement has a get-label-type
job at the top calling _runner-determinator.yml, then passes its outputs into the
build/test jobs:
jobs:
get-label-type:
name: get-label-type
uses: pytorch/pytorch/.github/workflows/_runner-determinator.yml@main
with:
triggering_actor: ${{ github.triggering_actor }}
issue_owner: ${{ github.event.pull_request.user.login || github.event.issue.user.login }}
curr_branch: ${{ github.head_ref || github.ref_name }}
curr_ref_type: ${{ github.ref_type }}
check_experiments: arc,lf
opt_out_experiments: lf
some-build:
uses: ./.github/workflows/_linux-build.yml
needs: get-label-type
with:
runner_prefix: "${{ needs.get-label-type.outputs.label-type }}"
runner: linux.c7i.2xlarge
use-arc: ${{ needs.get-label-type.outputs.use-arc == 'true' }}
test-matrix: |
{ include: [
{ config: "default", shard: 1, num_shards: 1, runner: "linux.aws.h100" },
]}
runner_prefix is empty by default — without the determinator interpolation, the
build lands on the default fleet (AWS EC2).
Experiment System
Config Source
The rollout config lives in the first comment of GitHub issue
pytorch/test-infra#5132. The script fetches it at runtime via the GitHub API.
The issue body has two ----separated sections:
- Settings YAML — defines available experiments and their rollout %.
- User opt-in list —
@username,experiment[:percent],-experiment_to_opt_out.
The issue number is overridable via the issue_number input on
_runner-determinator.yml (default "5132").
Experiment Settings (per the docstring example)
experiments:
lf:
rollout_percent: 25
all_branches: false
default: true
arc:
rollout_perc: 50
all_branches: true
default: false
workflows: pull,trunk
Per-experiment fields (Experiment NamedTuple in runner_determinator.py):
rollout_perc: float — % of workflows that get this experiment when no user opted in.all_branches: bool — if False, exception branches (release/main/etc.) skip the experiment.default: bool — if False, the experiment only runs when the caller passes it in check_experiments.workflows: str — comma-separated allowlist of github.workflow names. "ALL" or empty = every workflow. "-Name" prefix excludes that workflow even when "ALL" is present. Excludes win over includes.
User Opt-in / Opt-out
After the --- separator: each line @user,experiment1,experiment2:N,-experiment3:
- Plain entry → opt-in 100%.
experiment:N → per-user N% rollout (0-100).-experiment → explicit opt-out.#@user,... → user opts out of ALL experiments.- Triggering actor and PR author are both checked.
Per-Workflow Opt-In / Opt-Out (in the workflow YAML)
| Input | Effect |
|---|
check_experiments: arc,lf | Only consider these experiments (overrides default: true for non-listed ones — they will NOT run unless listed). |
check_experiments unset | Use the experiment's default: flag from the config. |
opt_out_experiments: lf | Explicitly skip the named experiment regardless of default: / check_experiments. Higher priority than check_experiments. |
PR-Level Kill Switch
Apply the no-runner-experiments label on a PR → determinator skips everything
and returns the default-fleet prefix (empty). See OPT_OUT_LABEL in
runner_determinator.py.
The Four Prefix States
| ARC | LF | label-type output | What lands on the build CPU runner |
|---|
| off | off | "" | linux.c7i.2xlarge |
| off | on | "lf." | lf.linux.c7i.2xlarge |
| on | off | "mt-" (= ARC_LABEL_PREFIX) | mt-linux.c7i.2xlarge -> via map_ec2_to_arc.py -> mt-l-x86iavx512-8-64 |
| on | on | "lf-" (after PR #186663) | lf-linux.c7i.2xlarge -> via map_ec2_to_arc.py -> lf-l-x86iavx512-8-64 |
Notes:
- ARC takes precedence over LF when both are enabled (returns
RunnerPrefixResult(use_arc=True)).
- Before PR #186663: ARC+LF returned
"mt-". After: ARC+LF returns "lf-", but still
sets use_arc=True. This routes jobs to LF-fleet ARC runners while keeping ARC's
scheduling semantics.
- The canary repo (
pytorch/pytorch-canary) appends .c to the LF prefix:
"lf.c." standalone, "c-mt-" for ARC standalone.
amd-do experiment is exposed separately via the amd-do-label-type output
("amd-do-"), not folded into label-type. Consumers wire it per-job.- Only one non-fleet experiment can stack with one fleet at a time (the rest are
dropped with a warning).
EC2 -> ARC Label Translation
When ARC is enabled (use-arc: true), the test job runs test-osdc (the OSDC branch
in _linux-test.yml) which needs ARC fleet labels, not EC2 labels. The translation
happens in the build job, not the test job:
_linux-build.yml runs map_ec2_to_arc.py --prefix "${RUNNER_PREFIX}" "${FILTERED_TEST_MATRIX}".- The script:
- Loads
runner_mapping from .github/arc.yaml.
- For each
include[].runner: strips the prefix, looks the bare label up in
runner_mapping, prepends the prefix back (with passthrough rules).
- Drops entries with config in
excluded_configs (currently {"onnx"} — see
the comment in the script for why).
- The translated test-matrix becomes the build job's
test-matrix output.
- The test job consumes
needs.<build>.outputs.test-matrix as its input, so
matrix.runner is the already-translated ARC label.
arc.yaml — The Mapping Table
runner_mapping:
linux.c7i.2xlarge: l-x86iavx512-8-64
linux.12xlarge.memory: l-x86iavx512-48-384
linux.aws.h100: l-x86iamx-22-225-h100
linux.aws.h100.4: l-x86iamx-88-900-h100-4
linux.aws.h100.8: l-bx86iamx-176-1800-h100-8
linux.dgx.b200: l-x86iamx-22-225-b200
linux.rocm.gpu.2: linux.rocm.gpu.2
linux.idc.xpu: linux.idc.xpu
Passthrough rule (identity mapping in the table): mapped == clean means
the runner is not OSDC-managed (e.g. ROCm, XPU, TPU). The script keeps the
original label without prefixing.
ARC label naming convention is documented as a comment at the top of arc.yaml:
{os}-[b]{arch}{vendor}{features}-{vcpu}-{memory}[-{gpu_type}[-{gpu_count}]]
Examples: l-x86iavx512-8-64 (Linux, x86, Intel AVX-512, 8 vCPU, 64 GiB),
l-bx86iamx-176-1800-h100-8 (Linux, bare-metal, x86, Intel AMX, 176 vCPU,
1800 GiB, 8x H100).
Meta-Only Override for H100/B200
H100 and B200 hardware only exists on the Meta ARC fleet — LF and AWS EC2 don't carry
those machines. Routing an H100/B200 job to lf-l-...-h100 would queue forever.
The fix: meta_only_runners list in .github/arc.yaml:
meta_only_runners:
- linux.aws.h100
- linux.aws.h100.4
- linux.aws.h100.8
- linux.dgx.b200
In map_ec2_to_arc.py, the per-entry loop has an early-return for any label in this
set: entry["runner"] = "mt-" + mapped (forces Meta ARC fleet, overrides whatever
--prefix was passed). This decouples the H100/B200 test runner from the build
runner's fleet — the build CPU is free to land anywhere the experiment routes it,
but the GPU test job is always pinned to Meta ARC.
Important: linux.dgx.b200.8 (8-GPU B200, used by b200-distributed.yml and
b200-symm-mem.yml) is NOT in runner_mapping today. Those workflows do NOT set
use-arc: true, so map_ec2_to_arc.py is never called for them — they land on AWS
EC2 directly. If you ever migrate them to ARC, the correct ARC label must be added
to BOTH runner_mapping AND meta_only_runners.
Build vs Test Runner Decoupling — Why It Matters
A single _runner-determinator.yml call typically feeds BOTH the build job's
runner_prefix AND the build job's use-arc (forwarded to the test job). They are
coupled by default. There are three patterns in the wild:
- Fully coupled (default) —
runner_prefix: and use-arc: both come from the
same get-label-type. Build CPU runner and test GPU runner travel together
through the experiment combinations.
- Hardcoded
mt- (runner_prefix: "mt-") — bypasses the determinator's output
for that specific build job. Used by attention_op_microbenchmark.yml and
operator_microbenchmark.yml for their B200 build paths. Comment in those files:
"always use OSDC runner to test this workflow".
- Per-workflow opt-out (
opt_out_experiments: lf) — used by
inductor-perf-test-nightly-h100.yml, inductor-perf-test-b200.yml,
inductor-periodic.yml. Removes a single experiment from consideration for that
workflow.
The meta_only_runners mechanism is the only decoupling that targets a specific
runner label rather than a whole workflow — it's the cleanest tool when "the build
can land anywhere but this specific test runner must always be Meta ARC".
GPU Workflows: Inventory and Mechanics
The workflows that touch high-end NVIDIA GPUs in this repo (H100, B200, A100). Read
each before editing — runner conventions differ.
| File | GPU test runners | check_experiments | opt_out_experiments |
|---|
.github/workflows/test-h100.yml | linux.aws.h100 | arc,lf | — |
.github/workflows/h100-cutlass-backend.yml | linux.aws.h100 | arc,lf | — |
.github/workflows/h100-distributed.yml | linux.aws.h100.8 | arc,lf | — |
.github/workflows/h100-symm-mem.yml | linux.aws.h100.4 | arc,lf | — |
.github/workflows/test-b200.yml | linux.dgx.b200 | arc,lf | — |
.github/workflows/b200-distributed.yml | linux.dgx.b200.8 | (none) | — |
.github/workflows/b200-symm-mem.yml | linux.dgx.b200.8 | (none) | — |
.github/workflows/inductor-perf-test-nightly-h100.yml | linux.aws.h100 | arc | lf |
.github/workflows/inductor-perf-test-b200.yml | linux.dgx.b200 | (none) | lf |
.github/workflows/inductor-pallas.yml | linux.aws.h100 | arc | — |
.github/workflows/inductor-periodic.yml | linux.aws.h100 (1 entry) + many g5/a100 | arc,amd-do | lf |
.github/workflows/attention_op_microbenchmark.yml | linux.aws.a100, linux.aws.h100, linux.dgx.b200 (B200 path: runner_prefix: "mt-" hardcoded) | arc,lf | — |
.github/workflows/operator_microbenchmark.yml | linux.aws.h100, linux.aws.a100, linux.dgx.b200 (B200 path: runner_prefix: "mt-" hardcoded) | arc,lf | — |
.github/workflows/operator_microbenchmark_compare.yml | conditional H100/A100/B200 via ${{ inputs.gpu }} | (none, on the get-label-type job) | — |
.github/workflows/vllm.yml | linux.dgx.b200 (1 entry) + many g6 | NO get-label-type job | — |
.github/workflows/vllm-benchmark.yml | inputs.runners default "h100,b200" → external matrix gen | NO get-label-type job | — |
Common Runner Label Families
CPU build runners (default fleet, prefixed by experiment)
linux.c7i.2xlarge, linux.4xlarge, linux.12xlarge, linux.12xlarge.memory,
linux.r7i.4xlarge, linux.24xlarge.memory, linux.24xl.spr-metal. These flow
through arc.yaml -> l-x86iavx512-* family.
NVIDIA GPU test runners
| Label | GPU |
|---|
linux.g4dn.4xlarge.nvidia.gpu, linux.g4dn.12xlarge.nvidia.gpu, linux.g4dn.metal.nvidia.gpu | T4 |
linux.g5.4xlarge.nvidia.gpu, linux.g5.12xlarge.nvidia.gpu, linux.g5.48xlarge.nvidia.gpu | A10G |
linux.g6.4xlarge.experimental.nvidia.gpu, linux.g6.12xlarge.nvidia.gpu | L4 |
linux.aws.a100 | A100 (p4de) |
linux.aws.h100, linux.aws.h100.4, linux.aws.h100.8 | H100 (p5) |
linux.dgx.b200, linux.dgx.b200.8 | B200 (p6) |
ARM64
linux.arm64.2xlarge, linux.arm64.m7g.4xlarge, linux.arm64.m8g.4xlarge,
linux.arm64.r7g.12xlarge.memory, linux.arm64.m7g.metal.
Partner hardware (passthrough — never prefixed/translated)
linux.rocm.gpu.*, linux.idc.xpu, linux.google.tpuv7x.1.
_runner-determinator.yml Outputs (Full Reference)
| Output | What it is | Typical consumer |
|---|
label-type | The runner prefix ("", "lf.", "mt-", "lf-") | runner_prefix: input on _linux-build.yml |
use-arc | String "true" / "false" | use-arc: input on _linux-build.yml / _linux-test.yml, often compared with == 'true' |
amd-do-label-type | "amd-do-" if amd-do experiment enabled, else "" | Per-job AMD pinning |
ci-docker-hash | git rev-parse HEAD:.ci/docker (the tree hash) | ci-docker-hash: input on _linux-build.yml for image tag |
runner-config | Normalized runner config (m7g / m8g) | ARM64 metal runner selection in some workflows |
runner-type | Runner suffix (metal / metal-24xl) | ARM64 metal runner selection |
runner-label | Fully qualified ARM label linux.arm64.<config>.<type> | Direct runs-on: on some ARM workflows |
Decision Trees / Recipes
"I want this workflow to never use LF runners"
Add opt_out_experiments: lf to the _runner-determinator.yml caller. The build's
runner_prefix will never include lf. / lf-. Existing examples:
inductor-perf-test-nightly-h100.yml, inductor-perf-test-b200.yml,
inductor-periodic.yml.
"I want this specific GPU test runner to always go to Meta ARC, but let the build follow the experiment"
Add the EC2 label to meta_only_runners in .github/arc.yaml. No workflow changes
needed. Already done for linux.aws.h100, linux.aws.h100.4, linux.aws.h100.8,
linux.dgx.b200.
"I want this build job to always go to Meta ARC (skip the determinator)"
Hardcode runner_prefix: "mt-" on the build job. Skips the experiment system for
that specific job. Already done for B200 build paths in
attention_op_microbenchmark.yml and operator_microbenchmark.yml.
"I want to know what label my job will actually run on"
Trace through:
- Read the workflow YAML — find the
get-label-type job and the build job.
- Note the
check_experiments / opt_out_experiments settings.
- Pull the current test-infra #5132 config (first comment) to see what the
experiments are currently rolling.
- Apply the prefix to the bare
runner label.
- If
use-arc=true, translate via arc.yaml.
- If the bare label is in
meta_only_runners, the result is mt-<arc-label>
regardless of step 4.
"I added a new EC2 runner type — what do I need to update?"
If it will EVER be used with use-arc: true: add the EC2 -> ARC mapping to
runner_mapping in .github/arc.yaml. If it's a passthrough (not OSDC-managed),
add an identity mapping (linux.foo: linux.foo). Otherwise the script errors out
with error: no ARC runner found for '<label>'.
If it's a new H100/B200 variant: also add the EC2 label to meta_only_runners.
"How do I add a new experiment?"
The experiment is defined entirely in the test-infra #5132 issue body. No code
change to runner_determinator.py is needed unless the experiment requires a
special handling (like arc returning use_arc=True, or amd-do being exposed
separately). Workflows opt in via check_experiments: <name> on their
get-label-type job.
Common Failure Modes
| Symptom | Likely cause |
|---|
Job stuck queued on lf-l-...-h100 or lf-l-...-b200 | H100/B200 not in meta_only_runners. Add it. |
map_ec2_to_arc.py error error: no ARC runner found for '<label>' | Missing entry in runner_mapping in .github/arc.yaml. Add a mapping (or an identity passthrough if not OSDC-managed). |
Job lands on linux.aws.h100 (EC2) instead of mt-l-x86iamx-22-225-h100 (ARC) | The build job didn't pass use-arc: true to the test job, so the test-matrix wasn't translated. Check the use-arc: line on the test job. |
Job lands on the default fleet (no prefix) despite check_experiments: arc,lf | Either the user is opted out via #@user in test-infra #5132, the PR has the no-runner-experiments label, or both experiments are below their rollout %. Set the user to 100% opt-in to debug. |
Workflow newly inherits lf- prefix after PR #186663 | Expected: arc,lf enabled together now yields lf- (was mt- before). If LF doesn't carry that hardware, opt out of LF or add the bare label to meta_only_runners. |
| New runner added to a test-matrix breaks the build job | Probably missing from arc.yaml. The script's per-entry lookup is strict — there is no implicit passthrough for unknown labels. |
| Onnx tests get silently dropped on the ARC path | Intentional. map_ec2_to_arc.py excludes config: onnx because onnxruntime's hardware_concurrency() sees all host CPUs on ARC k8s instead of the container cpuset. See the comment in the script. |
| Determinator script fails or times out | The wrapper catches the exception and falls back to Meta runners (empty prefix) + no experiments. Look in the get-label-type step logs for the actual error. |
check_experiments typo / unknown experiment name | The experiment is silently ignored (not in the eligible set). Check spelling against the keys in the test-infra #5132 settings YAML. |
--workflow-name not recognized by an older script | _runner-determinator.yml probes --help and skips the flag if the checked-out script (from PR head) doesn't support it. Workflow allowlist filtering won't run. Rebase the PR onto main. |
Things to NEVER Do
- NEVER invent a new ARC label format — use the existing
{os}-[b]{arch}{vendor}{features}-{vcpu}-{memory}[-{gpu_type}[-{gpu_count}]] scheme in arc.yaml. The convention is reviewed across CI infra.
- NEVER add an EC2 label to
meta_only_runners without first confirming it has a runner_mapping entry. The script will error.
- NEVER remove
use_arc=True from the ARC branch in runner_determinator.py — it's load-bearing for routing the test job to the test-osdc branch in _linux-test.yml.
- NEVER hardcode an ARC label (e.g.
l-x86iamx-22-225-h100) in a workflow's test-matrix runner: field. Use the EC2 label and let map_ec2_to_arc.py translate. map_ec2_to_arc.py does not recognize ARC labels as inputs.
- NEVER add the
lf- prefix or mt- prefix directly to runner_mapping entry values. The prefix is added separately by the script.
- NEVER edit
.ci/docker/ to change runner behavior — that's a Docker image rebuild trigger, unrelated to runner routing.
Key Cross-References
pytorch/test-infra#5132 — first comment is the live rollout config. Edit there to roll experiments, not in code.- PR #186663 — changes ARC+LF combination to return
"lf-" instead of "mt-". Read this when debugging why a workflow's prefix flipped after merging.
docs/runner_naming_convention.md (in ~/meta/ci-infra/osdc/) — the source of truth for the ARC label naming scheme. The osdc-runners-nodepools skill covers the runner definition side (what runners exist, what hardware they map to).~/meta/ci-infra/osdc/modules/arc-runners/defs/*.yaml, modules/arc-runners-h100/defs/, modules/arc-runners-b200/defs/ — actual ARC runner scale set definitions. Each runner.name field there is what appears as the bare label in arc.yaml's mapping target.
Verification Commands
gh issue view 5132 --repo pytorch/test-infra --json comments --jq '.comments[0].body'
cd ~/meta/pytorch
python .github/scripts/runner_determinator.py --help
cd ~/meta/pytorch/.github/scripts
python map_ec2_to_arc.py --prefix "lf-" '{include:[{config:"x",runner:"linux.aws.h100"}]}'
python .github/scripts/test_runner_determinator.py
python .github/scripts/test_map_ec2_to_arc.py
