# Manage Wheelhouse Components

> Upgrade a pinned component version or add a brand-new CUDA wheel component to the verl-wheelhouse build pipeline. Covers editing versions.yaml, writing ci/build_scripts/<builder>.sh, and creating/copying a .github/workflows/build-<component>.yml trigger workflow. Use when asked to bump/update/pin a submodule version, add a new wheel/component to the wheelhouse, or edit versions.yaml in this repo.

- Skill: `verl-project/manage-wheelhouse-components` (Agent Skill)
- Install (CLI): `npx skillmds@latest add verl-project/manage-wheelhouse-components`
- Raw SKILL.md: https://api.skillmd.com/api/skills/verl-project/manage-wheelhouse-components/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: verl-project (https://skillmd.com/u/verl-project)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/verl-project/manage-wheelhouse-components

---


# Managing verl-wheelhouse components

Everything is driven by `versions.yaml` (repo root, plain YAML data). For
the full step-by-step guide with a worked example, see
[docs/maintaining-components.md](../../../docs/maintaining-components.md).
This file has the quick-reference version.

## Upgrading an existing component's version

1. Edit the component's `ref:` under `components:` in `versions.yaml`.
2. If upstream changed supported GPU architectures, update
   `torch_cuda_arch_list` too (see arch-list conventions below).
3. Validate: `pip install pyyaml && python3 ci/generate_matrix.py --component <name>`.
4. Commit. Pushing to `main` auto-triggers `build-<component>.yml` (its
   `paths:` filter matches `versions.yaml`), and a successful build from a
   push also creates that component's per-Python persistent releases -
   one tag per interpreter (`<component>-<new-ref>` for legacy 3.12,
   `<component>-<new-ref>-pyX.Y` otherwise; see "Release naming" below and
   `ci/release_meta.py`) - uploads each wheel to its matching release, and
   republishes the index - no tag push needed to make it `pip
   install`-able. The previous ref's releases are left untouched as
   history.

`apex` tracks `main` unpinned (matches verl's own Dockerfiles) - there is no
version to bump for it.

## Adding a brand-new component

Checklist:

- [ ] `git submodule add <url> <path>`
- [ ] Add a `components:` entry to `versions.yaml` (`path`, `ref`,
      `builder`, `wheel_packages`, `torch_cuda_arch_list`, `requires_cudnn`,
      `max_jobs`, `runs_on`, `arches`, `extra_env`) - follow the schema
      comments already in that file. `wheel_packages` must list every
      distribution the build uploads; push builds use it to detect a complete
      matching per-Python release. Start with `arches: [x86_64]` and add arm64
      later
      (see below). Leave `python_versions` unset to build for every
      interpreter in the matrix (3.11 + 3.12); pin `python_versions: ["3.12"]`
      only for `py3-none-any` wheels or upstream `requires-python >=3.12`
      components (see "Python versions" below).
- [ ] Create `ci/build_scripts/<builder>.sh`. Copy the shape of an existing
      script (`ci/build_scripts/apex.sh` is a good default): shebang,
      `set -euo pipefail`, source `common.sh`, call `export_extra_env`,
      install prerequisite pip packages, run the project's own documented
      wheel-build command (mirror its own CI/Dockerfile exactly), leave the
      wheel(s) in `dist/` relative to CWD. Then `chmod +x` it.
- [ ] Copy an existing `.github/workflows/build-<component>.yml` to
      `build-<new-component>.yml`; update its `name:`, the `paths:` filter
      entries, and the `--component <name>` argument. The release is
      created inline by `_build.yml` (keyed by the matrix row's
      `release_tag`, one release per component and Python version), so
      there is no per-workflow release job to wire up. Leave the reusable
      workflow calls' structure and the `publish-index` job
      untouched - they're otherwise component-agnostic. Start from
      `build-flashinfer.yml` for a GitHub-hosted component, or
      `build-apex.yml` for a self-hosted one (it also forwards the
      `BYTED_PROXY` secret).
- [ ] Update `README.md`'s component table and repo-layout listing.
- [ ] Validate:
      `pip install pyyaml && python3 ci/generate_matrix.py --component <name>`
      and `bash -n ci/build_scripts/<builder>.sh`.
- [ ] No changes needed in `build-all.yml` or `release.yml` - both already
      use `--component all` and pick up new components automatically.

## CPU arches (x86_64 / aarch64)

`build_matrix` entries carry an `arch` field (`uname -m` spelling). A
component builds every arch in the matrix unless it narrows that with
`arches: [...]`, and can replace any field per arch via `arch_overrides`:

```yaml
arch_overrides:
  aarch64:
    runs_on: ubuntu-24.04-arm # GitHub-hosted arm64; self-hosted is rejected
    torch_cuda_arch_list: "9.0;10.0" # arm64 CUDA hosts are GH200/GB200-class only
```

To enable arm64 for a component: drop its `arches: [x86_64]`, add the
`arch_overrides.aarch64` block above, and - if it emits any `py3-none-any`
wheel - make the build script skip that wheel when `$TARGET_ARCH` is not
`x86_64`, or both arches will upload the same asset name. `_build.yml`
asserts the runner's `uname -m` matches the row's arch. No other change is
needed; CUDA, cuDNN, NCCL and torch installation are already arch-aware.

Non-x86_64 rows must stay on GitHub-hosted runners: the only self-hosted
machine here is x86_64, so a self-hosted arm64 row would queue forever.
`ci/generate_matrix.py` fails the matrix rather than emitting one. Expect
much lower `max_jobs` on the arm row (4 vCPU) and several resumed attempts
for the heavy builds.

`flash-attention`, `apex`, `transformer-engine`, `flashinfer`, `deep-ep`,
`flash-mla` and `fast-hadamard-transform` build both arches;
`megatron-bridge` is x86_64-only because its `py3-none-any` wheel already
covers arm64.

Each arch is a separate job. To build just one, dispatch
`build-<component>.yml` with its `arch` input (or pass
`--arch <arch>` to `ci/generate_matrix.py` locally); pushes always build
every arch. When adding a new arch to `build_matrix`, extend the static
`options:` list of every workflow's `arch` dispatch input to match.

## Python versions (3.11 / 3.12)

`build_matrix` rows carry a quoted `python: "3.11"` / `"3.12"` field. A
component builds every Python version in the matrix unless it narrows that
with `python_versions: [...]`, the exact counterpart of `arches`. The six
CUDA-extension components build both interpreters (their wheels carry
`cp311`/`cp312` tags); `megatron-bridge` and `flashinfer` pin
`python_versions: ["3.12"]` because their wheels are portable
`py3-none-any` (a second build only re-uploads an identical asset) and
Megatron-Bridge's upstream `requires-python` (">=3.12,<3.13") excludes 3.11.

To add a Python version:

- [ ] Verify torch publishes `cp<abi>` cu<cuda> wheels on **every arch** the
      component builds (`curl -sI https://download.pytorch.org/whl/cu130/...`).
- [ ] Append one quoted `python: "<v>"` row per arch to `build_matrix`.
- [ ] Extend the static `options:` list of the `python` dispatch input in
      **every** `.github/workflows/build-*.yml` (choice lists can't be
      generated; `generate_matrix.py` rejects unknown versions).
- [ ] Pin `python_versions: [...]` on any `py3-none-any` / `requires-python`
      component so it doesn't rebuild or fail.
- [ ] Validate:
      `python3 ci/generate_matrix.py --component all` (expect 27 rows today),
      `--component <name> --python <v>` for the smoke subset, and
      `cd ci && python3 -m unittest test_generate_matrix`.

To build just one interpreter, dispatch with the `python` input (or pass
`--python <v>` locally); an opted-out component yields an empty matrix and
its build job is skipped. Each interpreter publishes to its **own** release
(tag `<component>-<ref>-pyX.Y`, except the legacy 3.12 release, which keeps
the bare `<component>-<ref>` tag), and its title lists only that
interpreter's combos. Adding a version therefore only builds the new
interpreter's rows; existing interpreters' releases keep their tag/title
and stay skippable.

## Arch-list conventions

`torch_cuda_arch_list` in `versions.yaml` is canonical dotted+semicolon form
(e.g. `8.0;9.0;12.0`); each build script converts it as needed:

- apex: used as-is.
- flash-attention, TransformerEngine: undotted via `common.sh`'s
  `arch_list_strip_dots` (e.g. `80;90;120`).
- flashinfer: given verbatim with PTX-family suffixes (e.g.
  `8.0 9.0a 12.0f`) since those can't be derived mechanically.
- Megatron-Bridge, flash-mla, fast-hadamard-transform: set to `null` - the
  same goes for any component that builds no CUDA code, or that derives /
  hardcodes its own gencode flags instead of reading the variable.

## Release naming

Each component publishes to its **own** persistent GitHub Release **per
Python version** (no single combined release for the whole repo). The
legacy interpreter 3.12 keeps the bare tag `<component>-<ref>`; every
other interpreter uses `<component>-<ref>-pyX.Y` (e.g.
`<component>-<ref>-py3.11`). The title is
`<component> <ref> - [<arch> ]cu<cuda> py<python> torch<torch>[; ...]`
with one segment per `versions.yaml` `build_matrix` entry **for that
interpreter** (the `x86_64` arch left implicit), so a 3.11 release lists
only 3.11 rows. Tag, title, notes and manifest are computed by
`ci/release_meta.py` - which self-selects the running interpreter after
`setup-python` when `--python` is not passed - and the release is
created/refreshed inline by `_build.yml` on push builds. Don't hand-roll
`gh release create`/`edit` calls elsewhere.
`.github/workflows/_ensure_release.yml` is a legacy, currently uncalled
helper that only manages 3.12 bare-tag releases. Bumping a component's
`ref` starts brand-new releases under new tags; it never renames or reuses
the previous ref's.

## Key invariant

Every build script must leave the final `.whl` file(s) in `dist/` relative
to its own CWD (the component's checkout path) - `_build.yml` uploads
`<path>/dist/*.whl` as both a workflow artifact and (on release) a GitHub
Release asset.