| name | pytorch-runners-routing |
| description | How PyTorch CI jobs in .github/workflows/ resolve to a specific runner, AND which infrastructure that runner lives on: the two infras (legacy ALI Lambda-autoscaled EC2 vs current OSDC ARC-on-EKS) and the dot-vs-dash prefix split (linux./lf./c. = old EC2; mt-/lf-/c-mt- = OSDC ARC). Covers the runner determinator (fleet prefixes mt- default / lf- / c-mt- plus the separate amd-do-), the test-infra #5132 experiment config (lf, amd-do), the now-unconditional EC2->ARC label translation via .github/arc.yaml and map_ec2_to_arc.py (the meta_only_runners H100/B200 override and the onnx exclusion), build-vs-test runner decoupling, and the _runner-determinator.yml / _linux-build.yml / _linux-test.yml flow. Old-infra ownership spans test-infra/terraform-aws-github-runner, test-infra scale-config.yml / lf-scale-config.yml, and pytorch-gha-infra/runners; OSDC ownership lives in the osdc repo (clusters.yaml + modules/arc-runners*). Applies to ~/meta/pytorch. LOAD THIS SKILL for ANY question about PyTorch CI runners, runner labels, test matrices, runs-on values, runner prefixes (mt-, lf-, c-mt-, amd-do-, or legacy linux./lf./c.), WHICH infra a label runs on ("does lf.linux.2xlarge run on ARC or EC2?", "mt- vs lf- vs lf.?", "old ALI vs OSDC runners", "where is this runner defined?"), experiments (lf / amd-do), the determinator, ARC fleet routing, H100/B200/ A100 placement, or how a job gets assigned to a runner pool.
|
PyTorch CI Runner Routing & Resolution
Scope
Everything about how .github/workflows/*.yml jobs in pytorch/pytorch land on a
specific runner — label definition, which infrastructure serves it, fleet-prefix
resolution, the experiment system, and the EC2-to-ARC translation pipeline. Read this
BEFORE touching any runs-on: value, any test-matrix: entry, any runner_prefix:
interpolation, any check_experiments: / opt_out_experiments: input, arc.yaml, or
runner_determinator.py / map_ec2_to_arc.py.
For the legacy EC2 world in depth (the Lambda autoscaler, scale-config, dot prefixes,
who owns what), see references/old-ali-infra.md.
The Two Runner Infrastructures
Two completely different infrastructures serve PyTorch CI. Which one a job lands on is
encoded in its runner label's prefix.
- Old ALI — Lambda-autoscaled EC2 (legacy). A
workflow_job webhook drives a
scale-up Lambda that launches ONE ephemeral EC2 instance per job; the runner
self-registers with an exact-match label set from scale-config.yml, runs the job,
then is torn down. Owned by test-infra/terraform-aws-github-runner (the module),
test-infra:.github/scale-config.yml + lf-scale-config.yml (labels), and
pytorch-gha-infra/runners (Meta deploy). Being decommissioned; still serves
passthrough/identity labels. Depth in references/old-ali-infra.md.
- OSDC — ARC on EKS (current default). Actions Runner Controller schedules the
runner as a POD on a pre-existing EKS nodepool — no per-job VM boot. Owned by the
osdc repo:
clusters.yaml sets each cluster's runner_name_prefix, runner defs live
in modules/arc-runners*/defs/. Essentially every routed job lands here now.
The prefix tells you the infra. Dot/bare prefixes = old ALI EC2; dash prefixes =
OSDC ARC. The dot->dash shift is the ALI->OSDC migration (completed in pytorch/pytorch
PR #189219, which also removed the arc experiment and the old use-arc output).
Master prefix -> infrastructure map
Prefix in runs-on | Style | Infrastructure | Operator / funding | Defined / managed in |
|---|
mt- | dash | OSDC — ARC on EKS | Meta | osdc clusters.yaml meta-prod-aws-uw1/ue1/ue2; defs modules/arc-runners*/defs/. Determinator default (META_LABEL_PREFIX); also the error fallback. |
c-mt- | dash | OSDC — ARC on EKS | Meta (staging/canary) | osdc meta-staging-aws-*; META_CANARY_LABEL_PREFIX, emitted only on the pytorch/pytorch-canary repo. |
lf- | dash | OSDC — ARC on EKS | Linux Foundation | osdc clusters.yaml lf-prod-aws-ue1/ue2; LF_LABEL_PREFIX, emitted when the lf experiment is on. Same github.com/pytorch org — distinguished only by the scale-set-name prefix. |
amd-do- | dash | Partner ROCm (AMD dedicated) | AMD | separate amd-do-label-type output; identity-mapped in arc.yaml. |
linux. and other bare | bare | Old ALI — EC2 | Meta (gh-ci, AWS_PROFILE=fbossci) | test-infra:.github/scale-config.yml, deployed by pytorch-gha-infra. Legacy — determinator no longer emits it. map_ec2_to_arc.py translates ordinary bare linux.* -> mt-… (OSDC ARC); only arc.yaml identity/passthrough entries stay bare and land on literal old-infra/partner runners. |
lf. | dot | Old ALI — EC2 | Linux Foundation (parallel deploy) | test-infra:.github/lf-scale-config.yml. Dead in the determinator. |
Mental Model: Bare Label + Fleet Prefix + Unconditional Translation
For the Linux build/test path, the runs-on: value is assembled at runtime from:
- Bare EC2-style label in the workflow YAML (
linux.c7i.2xlarge, linux.aws.h100)
— what a human writes in the test-matrix.
- Fleet prefix from the determinator's
label-type output: mt- (default, Meta
OSDC), lf- (LF OSDC, when the lf experiment is on), or c-mt- (canary repo).
amd-do- is exposed separately via amd-do-label-type and wired per-job.
arc.yaml translation via map_ec2_to_arc.py — runs unconditionally on every
test-matrix, rewriting each EC2 label to its ARC equivalent (or leaving identity-
passthrough labels bare), and forcing mt- for meta_only_runners. There is no
use-arc toggle anymore; ARC is the destination for everything the determinator routes.
Examples: linux.c7i.2xlarge -> mt-l-x86iavx512-8-64 (default) or
lf-l-x86iavx512-8-64 (lf on). linux.aws.h100 -> mt-l-x86iamx-22-225-h100
regardless of prefix (H100/B200 pinned to Meta via meta_only_runners, see
Meta-Only Override for H100/B200).
linux.dgx.b200.8 stays linux.dgx.b200.8 (identity passthrough — old runner until
OSDC has capacity).
The 5 Files That Define Runner Routing
| File | Purpose |
|---|
.github/workflows/_runner-determinator.yml | Reusable workflow. Caller passes check_experiments / opt_out_experiments. Emits label-type (the fleet prefix), amd-do-label-type, runner-config, runner-type, runner-label, ci-docker-hash. |
.github/scripts/runner_determinator.py | The script. Fetches the rollout config from pytorch/test-infra#5132 (first comment), evaluates per-user opt-in/out + per-workflow allowlist + rollout %, and emits the fleet prefix. |
.github/scripts/test_runner_determinator.py | Tests for the determinator. Run on PR changes to the script. |
.github/arc.yaml | EC2 -> ARC label mapping (runner_mapping) AND meta_only_runners override list for H100/B200. |
.github/scripts/map_ec2_to_arc.py + test_map_ec2_to_arc.py | Script that rewrites a test-matrix's runner: field from EC2 labels to ARC labels using arc.yaml. Invoked unconditionally from _linux-build.yml. |
The Two Reusable Workflows That Consume the Determinator Output
| File | Role |
|---|
.github/workflows/_linux-build.yml | Build job. `runs-on: ${{ inputs.runner_prefix }}${{ startsWith(inputs.runner, 'l-') && inputs.runner |
.github/workflows/_linux-test.yml | Test job. A single job — runs-on: ${{ matrix.runner }} from the already-translated test-matrix, always in a container with options: "--gpus all", always assuming ARC IAM role arn:aws:iam::308535385114:role/arc. There is no longer a test vs test-osdc branch and no use-arc gate; the ARC path is unconditional. (setup-linux is called with a hardcoded use-arc: true input — that is an action input meaning "we are on ARC", unrelated to the deleted determinator output.) |
Sibling workflows exist for other platforms (_mac-build.yml, _mac-test.yml,
_win-build.yml, _win-test.yml, _rocm-test.yml, _xpu-test.yml, _vllm-build.yml,
_vllm-benchmark.yml, _linux-test-stable-fa3.yml). Not all participate in the
determinator/ARC flow — Linux build + test are the primary path.
Caller Pattern (How a Workflow Plugs Into the Determinator)
Every workflow that wants experiment-driven runner placement has a get-label-type
job at the top calling _runner-determinator.yml, then passes its outputs into the
build/test jobs:
jobs:
get-label-type:
name: get-label-type
uses: pytorch/pytorch/.github/workflows/_runner-determinator.yml@main
with:
triggering_actor: ${{ github.triggering_actor }}
issue_owner: ${{ github.event.pull_request.user.login || github.event.issue.user.login }}
curr_branch: ${{ github.head_ref || github.ref_name }}
curr_ref_type: ${{ github.ref_type }}
check_experiments: lf
some-build:
uses: ./.github/workflows/_linux-build.yml
needs: get-label-type
with:
runner_prefix: "${{ needs.get-label-type.outputs.label-type }}"
ci-docker-hash: ${{ needs.get-label-type.outputs.ci-docker-hash }}
runner: linux.c7i.2xlarge
The default fleet prefix is mt- (Meta OSDC) — the determinator never emits an empty
prefix. The build translates the test-matrix on every run and re-emits it; the test job
consumes needs.<build>.outputs.test-matrix, so matrix.runner is already the
translated ARC label. There is no use-arc: input to forward.
Experiment System
Config Source
The rollout config lives in the first comment of GitHub issue
pytorch/test-infra#5132. The script fetches it at runtime via the GitHub API.
The issue body has two ----separated sections:
- Settings YAML — defines available experiments and their rollout %.
- User opt-in list —
@username,experiment[:percent],-experiment_to_opt_out.
The issue number is overridable via the issue_number input on
_runner-determinator.yml (default "5132").
The live experiments are lf (switches the fleet to Linux Foundation OSDC) and
amd-do (routes ROCm mi350 tests to AMD's dedicated runners, exposed via the
separate amd-do-label-type output). Any other experiment name is still parsed, but if
it is neither lf nor amd-do the determinator logs "enabled but no longer affects the
runner label prefix; ignoring" and it changes nothing — there is no arc experiment
anymore.
Experiment Settings
experiments:
lf:
rollout_perc: 25
all_branches: false
default: true
Field-name caveat: the working field is rollout_perc (the Experiment NamedTuple
field). The docstring example inside runner_determinator.py writes rollout_percent —
a typo. parse_settings_from_text logs any unknown key as an "Unexpected setting" and
never applies it, so a config copied from that docstring gets rollout_perc = 0 (no
percentage rollout).
Per-experiment fields (Experiment NamedTuple in runner_determinator.py):
rollout_perc: float — % of workflows that get this experiment when no user opted in.all_branches: bool — if False, exception branches (main, nightly, release,
landchecks) skip the experiment.default: bool — if False, the experiment only runs when the caller passes it in
check_experiments.workflows: str — comma-separated allowlist of github.workflow names. "ALL" or
empty = every workflow. "-Name" prefix excludes that workflow even when "ALL" is
present. Exclusions win over inclusions.
User Opt-in / Opt-out
After the --- separator: each line @user,experiment1,experiment2:N,-experiment3:
- Plain entry -> opt-in 100%.
experiment:N -> per-user N% rollout (0-100).-experiment -> explicit opt-out.#@user,... -> user opts out of ALL experiments.- Triggering actor and PR author are both checked (the minimum per-user % among
opted-in requestors wins, so a conservative author % is respected).
Per-Workflow Opt-In / Opt-Out (in the workflow YAML)
| Input | Effect |
|---|
check_experiments: lf | Only consider these experiments (overrides default: true for non-listed ones — they will NOT run unless listed). |
check_experiments unset | Use each experiment's default: flag from the config (so lf is eligible by default). |
opt_out_experiments: lf | Explicitly skip the named experiment regardless of default: / check_experiments. Higher priority than check_experiments. |
PR-Level Kill Switch
Apply the no-runner-experiments label on a PR -> the determinator adds lf to the
opt-outs, so the run stays on the default Meta fleet (mt-) rather than LF. See
OPT_OUT_LABEL in runner_determinator.py.
Fleet Prefixes the Determinator Emits
| Condition | label-type output | Fleet / infra | Example test label (linux.c7i.2xlarge) |
|---|
Default (no lf) | mt- (META_LABEL_PREFIX) | Meta OSDC ARC | mt-l-x86iavx512-8-64 |
lf experiment on | lf- (LF_LABEL_PREFIX) | LF OSDC ARC | lf-l-x86iavx512-8-64 |
Canary repo (pytorch/pytorch-canary), no lf | c-mt- (META_CANARY_LABEL_PREFIX) | Meta OSDC ARC (staging) | c-mt-l-x86iavx512-8-64 |
| Determinator error / timeout | mt- (fallback) | Meta OSDC ARC | — |
PR has no-runner-experiments | mt- (opts out of lf) | Meta OSDC ARC | — |
Notes:
- The determinator can NEVER emit
"" (bare/EC2) or lf. (dotted). Those are
old-ALI prefixes — see references/old-ali-infra.md.
lf takes precedence over canary: on the canary repo with lf on, you get lf-,
not c-mt-.amd-do is exposed via its own amd-do-label-type output ("amd-do-" when enabled,
else ""), NOT folded into label-type. Consumers wire it per-job.
EC2 -> ARC Label Translation
The translation happens in the build job, unconditionally, so the test job receives
ARC fleet labels instead of EC2 labels:
_linux-build.yml runs map_ec2_to_arc.py --prefix "${RUNNER_PREFIX}" "${FILTERED_TEST_MATRIX}".- The script:
- Loads
runner_mapping and meta_only_runners from .github/arc.yaml.
- For each
include[].runner: strips the prefix, looks the bare label up in
runner_mapping, then re-prepends the prefix (with the passthrough and meta-only
rules below).
- Drops entries whose
config is in excluded_configs (currently {"onnx"} — see
Common Failure Modes).
- The translated test-matrix becomes the build job's
test-matrix output.
- The test job consumes
needs.<build>.outputs.test-matrix, so matrix.runner is the
already-translated ARC label.
arc.yaml — The Mapping Table
runner_mapping:
linux.c7i.2xlarge: l-x86iavx512-8-64
linux.12xlarge.memory: l-x86iavx512-48-384
linux.aws.h100: l-x86iamx-22-225-h100
linux.aws.h100.4: l-x86iamx-88-900-h100-4
linux.aws.h100.8: l-bx86iamx-176-1800-h100-8
linux.dgx.b200: l-x86iamx-22-225-b200
linux.dgx.b200.8: linux.dgx.b200.8
linux.rocm.gpu.2: linux.rocm.gpu.2
linux.idc.xpu: linux.idc.xpu
Passthrough rule (identity mapping in the table): mapped == clean means the runner
is not OSDC-managed (ROCm, XPU, TPU, and linux.dgx.b200.8 until OSDC has 8-GPU B200
capacity). The script keeps the original label WITHOUT prefixing — so it lands on the
existing self-hosted / old-infra runner.
ARC label naming convention is documented as a comment at the top of arc.yaml:
{os}-[b]{arch}{vendor}{features}-{vcpu}-{memory}[-{gpu_type}[-{gpu_count}]]
Examples: l-x86iavx512-8-64 (Linux, x86, Intel AVX-512, 8 vCPU, 64 GiB),
l-bx86iamx-176-1800-h100-8 (Linux, bare-metal, x86, Intel AMX, 176 vCPU, 1800 GiB,
8x H100).
Meta-Only Override for H100/B200
H100 and B200 hardware exists only on the Meta OSDC fleet — LF and AWS EC2 don't carry
those machines. Routing an H100/B200 job to lf-l-...-h100 would queue forever.
The fix: the meta_only_runners list in .github/arc.yaml:
meta_only_runners:
- linux.aws.h100
- linux.aws.h100.4
- linux.aws.h100.8
- linux.dgx.b200
In map_ec2_to_arc.py, the per-entry loop has an early branch for any label in this
set: entry["runner"] = "mt-" + mapped (forces Meta OSDC, overriding whatever
--prefix was passed). This decouples the H100/B200 test runner from the build runner's
fleet — the build CPU is free to land wherever the experiment routes it, but the GPU
test job is always pinned to Meta OSDC.
Important: linux.dgx.b200.8 (8-GPU B200, used by b200-distributed.yml and
b200-symm-mem.yml) is an identity passthrough in runner_mapping and is NOT in
meta_only_runners — OSDC has 8-GPU B200 runners but not enough capacity yet, so tests
stay on the existing linux.dgx.b200.8 runner while the OSDC build runs. When OSDC gains
capacity, replace the identity mapping with a real ARC label AND add the EC2 label to
meta_only_runners.
Build vs Test Runner Decoupling — Why It Matters
A single _runner-determinator.yml call typically feeds the build job's runner_prefix
(and its ci-docker-hash); the test runners come from the translated test-matrix. There
are four patterns in the wild:
- Fully coupled (default) —
runner_prefix: "${{ ...label-type }}". Build CPU and
the (translatable) test runners travel together through the experiment.
- Hardcoded
mt- (runner_prefix: "mt-") — pins that build job to Meta OSDC,
bypassing the determinator's fleet choice. Widely used for B200 builds, all ROCm
(rocm-*, periodic-rocm-*), XPU (xpu.yml), and several inductor perf builds.
- Per-workflow opt-out (
opt_out_experiments: lf) — removes lf from
consideration for that workflow. Used across the inductor family
(inductor.yml, inductor-unittest.yml, inductor-periodic.yml,
inductor-nightly.yml, inductor-perf-test-*), dynamo-unittest.yml, etc.
meta_only_runners — targets a specific runner label (H100/B200) rather than a
whole workflow. The cleanest tool when "the build can land anywhere but this specific
test runner must always be Meta OSDC".
GPU Workflows: Inventory and Mechanics
Workflows touching high-end NVIDIA GPUs (H100, B200, A100). check_experiments /
opt_out_experiments are re-derived from live source. (none) = the get-label-type
job passes no experiment inputs, so defaults apply (lf eligible unless opted out).
| File | GPU test runner(s) | check_experiments | opt_out_experiments | Notes |
|---|
test-h100.yml | linux.aws.h100 | lf | — | H100 forced mt- via meta_only_runners |
h100-cutlass-backend.yml | linux.aws.h100 | lf | — | |
h100-distributed.yml | linux.aws.h100.8 | lf | — | |
h100-symm-mem.yml | linux.aws.h100.4 | lf | — | |
test-b200.yml | linux.dgx.b200 | lf | — | B200 forced mt- via meta_only_runners |
b200-distributed.yml | linux.dgx.b200.8 | (none) | — | build pinned runner_prefix: "mt-"; get-label-type used only for ci-docker-hash; b200.8 test = identity passthrough -> old bare runner |
b200-symm-mem.yml | linux.dgx.b200.8 | (none) | — | same as b200-distributed.yml |
inductor-perf-test-nightly-h100.yml | linux.aws.h100 | (none) | lf | |
inductor-perf-test-b200.yml | linux.dgx.b200 | (none) | lf | build pinned runner_prefix: "mt-" |
inductor-pallas.yml | linux.aws.h100 | (none) | — |
Common Runner Label Families
CPU build runners (translated to the l-x86i* / l-arm64g* ARC families)
linux.c7i.2xlarge, linux.4xlarge, linux.12xlarge, linux.12xlarge.memory,
linux.r7i.4xlarge, linux.24xlarge.memory, linux.24xl.spr-metal, *.amx, *.avx2,
*.amd. The build job itself always lands on a fixed small ARC CPU (see
_linux-build.yml); these labels matter for the test matrix.
NVIDIA GPU test runners
| Label | GPU |
|---|
linux.g4dn.4xlarge.nvidia.gpu, linux.g4dn.12xlarge.nvidia.gpu, linux.g4dn.metal.nvidia.gpu | T4 |
linux.g5.4xlarge.nvidia.gpu, linux.g5.12xlarge.nvidia.gpu, linux.g5.48xlarge.nvidia.gpu | A10G |
linux.g6.4xlarge.experimental.nvidia.gpu, linux.g6.12xlarge.nvidia.gpu | L4 |
linux.aws.a100 | A100 (p4de) |
linux.aws.h100, linux.aws.h100.4, linux.aws.h100.8 | H100 (p5) — Meta-only |
linux.dgx.b200 | B200 (p6) — Meta-only |
linux.dgx.b200.8 | B200 8-GPU — identity passthrough (old runner) |
ARM64
linux.arm64.2xlarge, linux.arm64.m7g.4xlarge, linux.arm64.m8g.4xlarge,
linux.arm64.r7g.12xlarge.memory, linux.arm64.m7g.metal, linux.arm64.m8g.metal-24xl.
Partner hardware (identity passthrough — never prefixed/translated)
linux.rocm.gpu.2, linux.rocm.gpu.mi210.1/.2, linux.rocm.gpu.gfx942.1/.4,
linux.rocm.gpu.gfx950.1/.2, linux.rocm.gpu.gfx1100, linux.idc.xpu,
linux.client.xpu, linux.google.tpuv7x.1. The amd-do experiment routes mi350 tests
through the amd-do--prefixed identity entries (amd-do-linux.rocm.gpu.gfx950.1/.2).
_runner-determinator.yml Outputs (Full Reference)
| Output | What it is | Typical consumer |
|---|
label-type | The fleet prefix ("mt-", "c-mt-", "lf-") | runner_prefix: input on _linux-build.yml |
amd-do-label-type | "amd-do-" if the amd-do experiment is enabled, else "" | Per-job AMD pinning |
ci-docker-hash | git rev-parse HEAD:.ci/docker (the tree hash, from PR head) | ci-docker-hash: input on _linux-build.yml for the image tag |
runner-config | Normalized runner config (m7g / m8g) | ARM64 metal runner selection |
runner-type | Runner suffix (metal / metal-24xl) | ARM64 metal runner selection |
runner-label | Fully qualified ARM label linux.arm64.<config>.<type> | Direct runs-on: on some ARM workflows |
There is no use-arc output (removed with the arc experiment in PR #189219).
Decision Trees / Recipes
"I want this workflow to never use LF runners"
Add opt_out_experiments: lf to the _runner-determinator.yml caller. The prefix will
never be lf- — it stays mt-. Examples: the whole inductor family
(inductor.yml, inductor-unittest.yml, inductor-periodic.yml, inductor-nightly.yml,
inductor-perf-test-*), dynamo-unittest.yml.
"I want this specific GPU test runner to always go to Meta OSDC, but let the build follow the experiment"
Add the EC2 label to meta_only_runners in .github/arc.yaml. No workflow changes
needed. Already done for linux.aws.h100, linux.aws.h100.4, linux.aws.h100.8,
linux.dgx.b200.
"I want this build job to always go to Meta OSDC (skip the determinator's fleet choice)"
Hardcode runner_prefix: "mt-" on the build job. Skips the fleet experiment for that
job. Widely used for B200 builds, all ROCm, XPU, and some inductor perf builds.
"I want to know what infra + label my job will actually run on"
Trace through:
- Read the workflow YAML — find the
get-label-type job and the build/test jobs.
- Note
check_experiments / opt_out_experiments.
- Pull the current test-infra #5132 config (first comment) to see what's rolling.
- Determine the fleet prefix:
mt- unless lf is enabled (lf-) or it's the canary
repo (c-mt-).
- Apply
map_ec2_to_arc.py (it runs on EVERY matrix): EC2 label -> ARC label with the
prefix; identity-passthrough labels stay bare (old infra); meta_only_runners force
mt-.
- Dash prefix -> OSDC ARC; bare/dotted -> old ALI EC2 (see
references/old-ali-infra.md).
"I added a new EC2 runner type — what do I need to update?"
Add the EC2 -> ARC mapping to runner_mapping in .github/arc.yaml. If it's a
passthrough (not OSDC-managed), add an identity mapping (linux.foo: linux.foo).
Otherwise map_ec2_to_arc.py errors with error: no ARC runner found for '<label>' —
the lookup is strict, there is no implicit passthrough. If it's a new H100/B200 variant,
also add the EC2 label to meta_only_runners.
"How do I add a new experiment?"
Define it in the test-infra #5132 issue body. No code change to runner_determinator.py
is needed UNLESS the experiment must change the label prefix — the script only special-
cases lf (sets the fleet) and amd-do (exposed via amd-do-label-type). Any other
experiment is parsed but "no longer affects the runner label prefix" and does nothing to
routing. Workflows opt in via check_experiments: <name>.
Common Failure Modes
| Symptom | Likely cause |
|---|
Job stuck queued on lf-l-...-h100 or lf-l-...-b200 | H100/B200 not in meta_only_runners. Add it. |
map_ec2_to_arc.py error error: no ARC runner found for '<label>' | Missing entry in runner_mapping. Add a mapping (or an identity passthrough if not OSDC-managed). |
GPU test lands on a bare/old label instead of mt-l-... | The bare label is an identity passthrough in runner_mapping (e.g. linux.dgx.b200.8) — intentional until OSDC capacity lands. To move it to OSDC, replace the identity mapping and add a meta_only_runners entry. |
Job lands on the default Meta fleet despite expecting lf- | User opted out via #@user in #5132, PR has no-runner-experiments (opts out of lf), lf is below rollout %, or the workflow sets opt_out_experiments: lf. Set the user to 100% opt-in to debug. |
| New runner added to a test-matrix breaks the build's map step | Missing from arc.yaml. The per-entry lookup is strict — no implicit passthrough for unknown labels. |
| Onnx tests silently dropped on the ARC path | Intentional. map_ec2_to_arc.py excludes config: onnx because onnxruntime's hardware_concurrency() sees all host CPUs on ARC k8s instead of the container cpuset. See the comment in the script. |
| Determinator script fails or times out | The wrapper falls back to Meta runners (mt-) + no experiments. Look in the get-label-type step logs for the actual error. |
check_experiments typo / unknown experiment name | Silently ignored. Check spelling against the keys in the #5132 settings YAML (lf, amd-do). |
A stale determinator checked out from a PR emits lf. | _runner-determinator.yml checks the script out from the merge commit (base), not PR head, precisely so a stale prefix like lf. can't reach main's build job (which has no EC2 fallback). See the checkout comment citing #189113/#189171. |
Things to NEVER Do
- NEVER invent a new ARC label format — use the existing
{os}-[b]{arch}{vendor}{features}-{vcpu}-{memory}[-{gpu_type}[-{gpu_count}]] scheme in
arc.yaml. The convention is reviewed across CI infra.
- NEVER add an EC2 label to
meta_only_runners without first confirming it has a
runner_mapping entry. The script will error.
- NEVER hardcode an ARC label (e.g.
l-x86iamx-22-225-h100) in a workflow's
test-matrix runner: field. Use the EC2 label and let map_ec2_to_arc.py translate
it — the script only recognizes EC2/bare labels as input.
- NEVER add the
lf- / mt- prefix directly to runner_mapping entry values. The
prefix is added separately by the script.
- NEVER make the determinator emit a bare (
"") or dotted (lf.) prefix — main's
build/test path has no EC2 fallback, so an old-style prefix breaks routing (see
#189113/#189171).
- NEVER edit
.ci/docker/ to change runner behavior — that's a Docker image rebuild
trigger, unrelated to runner routing.
Key Cross-References
pytorch/test-infra#5132 — first comment is the live rollout config. Edit there to
roll experiments, not in code.references/old-ali-infra.md — the legacy Lambda-autoscaled EC2 world: scale-config,
dot prefixes, ownership, and how to recognize an old-ALI job today.- PR #189219 — removed the
arc experiment and the use-arc output; completed the
dot->dash (ALI->OSDC) migration. Read when confused why old docs mention use-arc.
docs/runner_naming_convention.md (in ~/meta/ci-infra/osdc/) — source of truth for
the ARC label naming scheme. The osdc-runners-nodepools skill covers the runner
definition side (what runners exist, what hardware they map to).~/meta/ci-infra/osdc/modules/arc-runners*/defs/*.yaml — the actual ARC runner scale
set definitions. Each runner.name there is what appears as the mapping target in
arc.yaml; runnerScaleSetName = {prefix}{runner.name} is the literal runs-on.
Verification Commands
gh issue view 5132 --repo pytorch/test-infra --json comments --jq '.comments[0].body'
cd ~/meta/pytorch
python .github/scripts/runner_determinator.py --help
cd ~/meta/pytorch/.github/scripts
uv run --no-project --with pyyaml==6.0.2 \
python map_ec2_to_arc.py --prefix "lf-" '{ include: [ { config: "x", runner: "linux.aws.h100" } ] }'
python .github/scripts/test_runner_determinator.py
python .github/scripts/test_map_ec2_to_arc.py
