Pytorch Runners Routing

PyTorch CI Runner Routing & Resolution

Scope

Everything about how .github/workflows/*.yml jobs in pytorch/pytorch land on a specific runner — label definition, which infrastructure serves it, fleet-prefix resolution, the experiment system, and the EC2-to-ARC translation pipeline. Read this BEFORE touching any runs-on: value, any test-matrix: entry, any runner_prefix: interpolation, any check_experiments: / opt_out_experiments: input, arc.yaml, or runner_determinator.py / map_ec2_to_arc.py.

For the legacy EC2 world in depth (the Lambda autoscaler, scale-config, dot prefixes, who owns what), see references/old-ali-infra.md.

The Two Runner Infrastructures

Two completely different infrastructures serve PyTorch CI. Which one a job lands on is encoded in its runner label's prefix.

• Old ALI — Lambda-autoscaled EC2 (legacy). A workflow_job webhook drives a scale-up Lambda that launches ONE ephemeral EC2 instance per job; the runner self-registers with an exact-match label set from scale-config.yml, runs the job, then is torn down. Owned by test-infra/terraform-aws-github-runner (the module), test-infra:.github/scale-config.yml + lf-scale-config.yml (labels), and pytorch-gha-infra/runners (Meta deploy). Being decommissioned; still serves passthrough/identity labels. Depth in references/old-ali-infra.md.
• OSDC — ARC on EKS (current default). Actions Runner Controller schedules the runner as a POD on a pre-existing EKS nodepool — no per-job VM boot. Owned by the osdc repo: clusters.yaml sets each cluster's runner_name_prefix, runner defs live in modules/arc-runners*/defs/. Essentially every routed job lands here now.

The prefix tells you the infra. Dot/bare prefixes = old ALI EC2; dash prefixes = OSDC ARC. The dot->dash shift is the ALI->OSDC migration (completed in pytorch/pytorch PR #189219, which also removed the arc experiment and the old use-arc output).

Master prefix -> infrastructure map

Mental Model: Bare Label + Fleet Prefix + Unconditional Translation

For the Linux build/test path, the runs-on: value is assembled at runtime from:

1. Bare EC2-style label in the workflow YAML (linux.c7i.2xlarge, linux.aws.h100) — what a human writes in the test-matrix.
2. Fleet prefix from the determinator's label-type output: mt- (default, Meta OSDC), lf- (LF OSDC, when the lf experiment is on), or c-mt- (canary repo). amd-do- is exposed separately via amd-do-label-type and wired per-job.
3. arc.yaml translation via map_ec2_to_arc.py — runs unconditionally on every test-matrix, rewriting each EC2 label to its ARC equivalent (or leaving identity- passthrough labels bare), and forcing mt- for meta_only_runners. There is no use-arc toggle anymore; ARC is the destination for everything the determinator routes.

Examples: linux.c7i.2xlarge -> mt-l-x86iavx512-8-64 (default) or lf-l-x86iavx512-8-64 (lf on). linux.aws.h100 -> mt-l-x86iamx-22-225-h100 regardless of prefix (H100/B200 pinned to Meta via meta_only_runners, see Meta-Only Override for H100/B200). linux.dgx.b200.8 stays linux.dgx.b200.8 (identity passthrough — old runner until OSDC has capacity).

The 5 Files That Define Runner Routing

The Two Reusable Workflows That Consume the Determinator Output

Sibling workflows exist for other platforms (_mac-build.yml, _mac-test.yml, _win-build.yml, _win-test.yml, _rocm-test.yml, _xpu-test.yml, _vllm-build.yml, _vllm-benchmark.yml, _linux-test-stable-fa3.yml). Not all participate in the determinator/ARC flow — Linux build + test are the primary path.

Caller Pattern (How a Workflow Plugs Into the Determinator)

Every workflow that wants experiment-driven runner placement has a get-label-type job at the top calling _runner-determinator.yml, then passes its outputs into the build/test jobs:

jobs:
  get-label-type:
    name: get-label-type
    uses: pytorch/pytorch/.github/workflows/_runner-determinator.yml@main
    with:
      triggering_actor: ${{ github.triggering_actor }}
      issue_owner: ${{ github.event.pull_request.user.login || github.event.issue.user.login }}
      curr_branch: ${{ github.head_ref || github.ref_name }}
      curr_ref_type: ${{ github.ref_type }}
      check_experiments: lf       # non-default experiments to consider for THIS workflow (optional)
      # opt_out_experiments: lf   # experiments to explicitly skip for THIS workflow (optional)

  some-build:
    uses: ./.github/workflows/_linux-build.yml
    needs: get-label-type
    with:
      runner_prefix: "${{ needs.get-label-type.outputs.label-type }}"   # fleet prefix
      ci-docker-hash: ${{ needs.get-label-type.outputs.ci-docker-hash }} # image tag suffix
      runner: linux.c7i.2xlarge                                          # base build label
      test-matrix: |
        { include: [
          { config: "default", shard: 1, num_shards: 1, runner: "linux.aws.h100" },
        ]}

The default fleet prefix is mt- (Meta OSDC) — the determinator never emits an empty prefix. The build translates the test-matrix on every run and re-emits it; the test job consumes needs.<build>.outputs.test-matrix, so matrix.runner is already the translated ARC label. There is no use-arc: input to forward.

Experiment System

Config Source

The rollout config lives in the first comment of GitHub issue pytorch/test-infra#5132. The script fetches it at runtime via the GitHub API. The issue body has two ----separated sections:

1. Settings YAML — defines available experiments and their rollout %.
2. User opt-in list — @username,experiment[:percent],-experiment_to_opt_out.

The issue number is overridable via the issue_number input on _runner-determinator.yml (default "5132").

The live experiments are lf (switches the fleet to Linux Foundation OSDC) and amd-do (routes ROCm mi350 tests to AMD's dedicated runners, exposed via the separate amd-do-label-type output). Any other experiment name is still parsed, but if it is neither lf nor amd-do the determinator logs "enabled but no longer affects the runner label prefix; ignoring" and it changes nothing — there is no arc experiment anymore.

Experiment Settings

experiments:
  lf:
    rollout_perc: 25
    all_branches: false
    default: true

Field-name caveat: the working field is rollout_perc (the Experiment NamedTuple field). The docstring example inside runner_determinator.py writes rollout_percent — a typo. parse_settings_from_text logs any unknown key as an "Unexpected setting" and never applies it, so a config copied from that docstring gets rollout_perc = 0 (no percentage rollout).

Per-experiment fields (Experiment NamedTuple in runner_determinator.py):

• rollout_perc: float — % of workflows that get this experiment when no user opted in.
• all_branches: bool — if False, exception branches (main, nightly, release, landchecks) skip the experiment.
• default: bool — if False, the experiment only runs when the caller passes it in check_experiments.
• workflows: str — comma-separated allowlist of github.workflow names. "ALL" or empty = every workflow. "-Name" prefix excludes that workflow even when "ALL" is present. Exclusions win over inclusions.

User Opt-in / Opt-out

After the --- separator: each line @user,experiment1,experiment2:N,-experiment3:

• Plain entry -> opt-in 100%.
• experiment:N -> per-user N% rollout (0-100).
• -experiment -> explicit opt-out.
• #@user,... -> user opts out of ALL experiments.
• Triggering actor and PR author are both checked (the minimum per-user % among opted-in requestors wins, so a conservative author % is respected).

Per-Workflow Opt-In / Opt-Out (in the workflow YAML)

PR-Level Kill Switch

Apply the no-runner-experiments label on a PR -> the determinator adds lf to the opt-outs, so the run stays on the default Meta fleet (mt-) rather than LF. See OPT_OUT_LABEL in runner_determinator.py.

Fleet Prefixes the Determinator Emits

Notes:

• The determinator can NEVER emit "" (bare/EC2) or lf. (dotted). Those are old-ALI prefixes — see references/old-ali-infra.md.
• lf takes precedence over canary: on the canary repo with lf on, you get lf-, not c-mt-.
• amd-do is exposed via its own amd-do-label-type output ("amd-do-" when enabled, else ""), NOT folded into label-type. Consumers wire it per-job.

EC2 -> ARC Label Translation

The translation happens in the build job, unconditionally, so the test job receives ARC fleet labels instead of EC2 labels:

1. _linux-build.yml runs map_ec2_to_arc.py --prefix "${RUNNER_PREFIX}" "${FILTERED_TEST_MATRIX}".
2. The script:
   • Loads runner_mapping and meta_only_runners from .github/arc.yaml.
   • For each include[].runner: strips the prefix, looks the bare label up in runner_mapping, then re-prepends the prefix (with the passthrough and meta-only rules below).
   • Drops entries whose config is in excluded_configs (currently {"onnx"} — see Common Failure Modes).
3. The translated test-matrix becomes the build job's test-matrix output.
4. The test job consumes needs.<build>.outputs.test-matrix, so matrix.runner is the already-translated ARC label.

arc.yaml — The Mapping Table

runner_mapping:
  linux.c7i.2xlarge: l-x86iavx512-8-64
  linux.12xlarge.memory: l-x86iavx512-48-384
  linux.aws.h100: l-x86iamx-22-225-h100
  linux.aws.h100.4: l-x86iamx-88-900-h100-4
  linux.aws.h100.8: l-bx86iamx-176-1800-h100-8
  linux.dgx.b200: l-x86iamx-22-225-b200
  # ... CPU, A100, H100, B200, A10G, T4, L4, ARM64
  linux.dgx.b200.8: linux.dgx.b200.8                   # passthrough — identity mapping
  linux.rocm.gpu.2: linux.rocm.gpu.2                   # passthrough
  linux.idc.xpu: linux.idc.xpu                         # passthrough

Passthrough rule (identity mapping in the table): mapped == clean means the runner is not OSDC-managed (ROCm, XPU, TPU, and linux.dgx.b200.8 until OSDC has 8-GPU B200 capacity). The script keeps the original label WITHOUT prefixing — so it lands on the existing self-hosted / old-infra runner.

ARC label naming convention is documented as a comment at the top of arc.yaml:

{os}-[b]{arch}{vendor}{features}-{vcpu}-{memory}[-{gpu_type}[-{gpu_count}]]

Examples: l-x86iavx512-8-64 (Linux, x86, Intel AVX-512, 8 vCPU, 64 GiB), l-bx86iamx-176-1800-h100-8 (Linux, bare-metal, x86, Intel AMX, 176 vCPU, 1800 GiB, 8x H100).

Meta-Only Override for H100/B200

H100 and B200 hardware exists only on the Meta OSDC fleet — LF and AWS EC2 don't carry those machines. Routing an H100/B200 job to lf-l-...-h100 would queue forever.

The fix: the meta_only_runners list in .github/arc.yaml:

meta_only_runners:
  - linux.aws.h100
  - linux.aws.h100.4
  - linux.aws.h100.8
  - linux.dgx.b200

In map_ec2_to_arc.py, the per-entry loop has an early branch for any label in this set: entry["runner"] = "mt-" + mapped (forces Meta OSDC, overriding whatever --prefix was passed). This decouples the H100/B200 test runner from the build runner's fleet — the build CPU is free to land wherever the experiment routes it, but the GPU test job is always pinned to Meta OSDC.

Important: linux.dgx.b200.8 (8-GPU B200, used by b200-distributed.yml and b200-symm-mem.yml) is an identity passthrough in runner_mapping and is NOT in meta_only_runners — OSDC has 8-GPU B200 runners but not enough capacity yet, so tests stay on the existing linux.dgx.b200.8 runner while the OSDC build runs. When OSDC gains capacity, replace the identity mapping with a real ARC label AND add the EC2 label to meta_only_runners.

Build vs Test Runner Decoupling — Why It Matters

A single _runner-determinator.yml call typically feeds the build job's runner_prefix (and its ci-docker-hash); the test runners come from the translated test-matrix. There are four patterns in the wild:

1. Fully coupled (default) — runner_prefix: "${{ ...label-type }}". Build CPU and the (translatable) test runners travel together through the experiment.
2. Hardcoded mt- (runner_prefix: "mt-") — pins that build job to Meta OSDC, bypassing the determinator's fleet choice. Widely used for B200 builds, all ROCm (rocm-*, periodic-rocm-*), XPU (xpu.yml), and several inductor perf builds.
3. Per-workflow opt-out (opt_out_experiments: lf) — removes lf from consideration for that workflow. Used across the inductor family (inductor.yml, inductor-unittest.yml, inductor-periodic.yml, inductor-nightly.yml, inductor-perf-test-*), dynamo-unittest.yml, etc.
4. meta_only_runners — targets a specific runner label (H100/B200) rather than a whole workflow. The cleanest tool when "the build can land anywhere but this specific test runner must always be Meta OSDC".

GPU Workflows: Inventory and Mechanics

Workflows touching high-end NVIDIA GPUs (H100, B200, A100). check_experiments / opt_out_experiments are re-derived from live source. (none) = the get-label-type job passes no experiment inputs, so defaults apply (lf eligible unless opted out).

Common Runner Label Families

CPU build runners (translated to the l-x86i* / l-arm64g* ARC families)

linux.c7i.2xlarge, linux.4xlarge, linux.12xlarge, linux.12xlarge.memory, linux.r7i.4xlarge, linux.24xlarge.memory, linux.24xl.spr-metal, *.amx, *.avx2, *.amd. The build job itself always lands on a fixed small ARC CPU (see _linux-build.yml); these labels matter for the test matrix.

NVIDIA GPU test runners

ARM64

linux.arm64.2xlarge, linux.arm64.m7g.4xlarge, linux.arm64.m8g.4xlarge, linux.arm64.r7g.12xlarge.memory, linux.arm64.m7g.metal, linux.arm64.m8g.metal-24xl.

Partner hardware (identity passthrough — never prefixed/translated)

linux.rocm.gpu.2, linux.rocm.gpu.mi210.1/.2, linux.rocm.gpu.gfx942.1/.4, linux.rocm.gpu.gfx950.1/.2, linux.rocm.gpu.gfx1100, linux.idc.xpu, linux.client.xpu, linux.google.tpuv7x.1. The amd-do experiment routes mi350 tests through the amd-do--prefixed identity entries (amd-do-linux.rocm.gpu.gfx950.1/.2).

_runner-determinator.yml Outputs (Full Reference)

There is no use-arc output (removed with the arc experiment in PR #189219).

Decision Trees / Recipes

"I want this workflow to never use LF runners"

Add opt_out_experiments: lf to the _runner-determinator.yml caller. The prefix will never be lf- — it stays mt-. Examples: the whole inductor family (inductor.yml, inductor-unittest.yml, inductor-periodic.yml, inductor-nightly.yml, inductor-perf-test-*), dynamo-unittest.yml.

"I want this specific GPU test runner to always go to Meta OSDC, but let the build follow the experiment"

Add the EC2 label to meta_only_runners in .github/arc.yaml. No workflow changes needed. Already done for linux.aws.h100, linux.aws.h100.4, linux.aws.h100.8, linux.dgx.b200.

"I want this build job to always go to Meta OSDC (skip the determinator's fleet choice)"

Hardcode runner_prefix: "mt-" on the build job. Skips the fleet experiment for that job. Widely used for B200 builds, all ROCm, XPU, and some inductor perf builds.

"I want to know what infra + label my job will actually run on"

Trace through:

1. Read the workflow YAML — find the get-label-type job and the build/test jobs.
2. Note check_experiments / opt_out_experiments.
3. Pull the current test-infra #5132 config (first comment) to see what's rolling.
4. Determine the fleet prefix: mt- unless lf is enabled (lf-) or it's the canary repo (c-mt-).
5. Apply map_ec2_to_arc.py (it runs on EVERY matrix): EC2 label -> ARC label with the prefix; identity-passthrough labels stay bare (old infra); meta_only_runners force mt-.
6. Dash prefix -> OSDC ARC; bare/dotted -> old ALI EC2 (see references/old-ali-infra.md).

"I added a new EC2 runner type — what do I need to update?"

Add the EC2 -> ARC mapping to runner_mapping in .github/arc.yaml. If it's a passthrough (not OSDC-managed), add an identity mapping (linux.foo: linux.foo). Otherwise map_ec2_to_arc.py errors with error: no ARC runner found for '<label>' — the lookup is strict, there is no implicit passthrough. If it's a new H100/B200 variant, also add the EC2 label to meta_only_runners.

"How do I add a new experiment?"

Define it in the test-infra #5132 issue body. No code change to runner_determinator.py is needed UNLESS the experiment must change the label prefix — the script only special- cases lf (sets the fleet) and amd-do (exposed via amd-do-label-type). Any other experiment is parsed but "no longer affects the runner label prefix" and does nothing to routing. Workflows opt in via check_experiments: <name>.

Common Failure Modes

Things to NEVER Do

• NEVER invent a new ARC label format — use the existing {os}-[b]{arch}{vendor}{features}-{vcpu}-{memory}[-{gpu_type}[-{gpu_count}]] scheme in arc.yaml. The convention is reviewed across CI infra.
• NEVER add an EC2 label to meta_only_runners without first confirming it has a runner_mapping entry. The script will error.
• NEVER hardcode an ARC label (e.g. l-x86iamx-22-225-h100) in a workflow's test-matrix runner: field. Use the EC2 label and let map_ec2_to_arc.py translate it — the script only recognizes EC2/bare labels as input.
• NEVER add the lf- / mt- prefix directly to runner_mapping entry values. The prefix is added separately by the script.
• NEVER make the determinator emit a bare ("") or dotted (lf.) prefix — main's build/test path has no EC2 fallback, so an old-style prefix breaks routing (see #189113/#189171).
• NEVER edit .ci/docker/ to change runner behavior — that's a Docker image rebuild trigger, unrelated to runner routing.

Key Cross-References

• pytorch/test-infra#5132 — first comment is the live rollout config. Edit there to roll experiments, not in code.
• references/old-ali-infra.md — the legacy Lambda-autoscaled EC2 world: scale-config, dot prefixes, ownership, and how to recognize an old-ALI job today.
• PR #189219 — removed the arc experiment and the use-arc output; completed the dot->dash (ALI->OSDC) migration. Read when confused why old docs mention use-arc.
• docs/runner_naming_convention.md (in ~/meta/ci-infra/osdc/) — source of truth for the ARC label naming scheme. The osdc-runners-nodepools skill covers the runner definition side (what runners exist, what hardware they map to).
• ~/meta/ci-infra/osdc/modules/arc-runners*/defs/*.yaml — the actual ARC runner scale set definitions. Each runner.name there is what appears as the mapping target in arc.yaml; runnerScaleSetName = {prefix}{runner.name} is the literal runs-on.

Verification Commands

# See the current test-infra #5132 config
gh issue view 5132 --repo pytorch/test-infra --json comments --jq '.comments[0].body'

# Run the determinator locally (read-only, hits the GitHub API)
cd ~/meta/pytorch
python .github/scripts/runner_determinator.py --help

# Run the ARC mapping for a test-matrix (matches the _linux-build.yml uv invocation)
cd ~/meta/pytorch/.github/scripts
uv run --no-project --with pyyaml==6.0.2 \
  python map_ec2_to_arc.py --prefix "lf-" '{ include: [ { config: "x", runner: "linux.aws.h100" } ] }'
# Expected: runner becomes "mt-l-x86iamx-22-225-h100" (meta_only override forces mt-)

# Run the test suites
python .github/scripts/test_runner_determinator.py
python .github/scripts/test_map_ec2_to_arc.py