NVIDIA-NeMo CI/CD Guide
How CI Works
NeMo-RL CI runs on GitHub Actions. Workflows live in .github/workflows/.
The main test workflow triggers on pushes to pull-request/<number> branches.
These branches are created automatically by copy-pr-bot when a contributor
pushes to their fork and the PR receives a trust signal:
- The commit is GPG-signed by a maintainer, or
- A maintainer posts
/ok to test <full-sha>as a PR comment.
Triggering CI
After pushing a new commit to your PR, trigger CI with:
/ok to test <full-commit-sha>
Use git rev-parse HEAD (not the short form) to get the full SHA.
Re-triggering after new commits: each
/ok to test <sha>is specific to that SHA. After pushing additional commits, post a new comment with the updated SHA.
CI Labels
Every PR must have exactly one CI:* label — the quality-check job stays
red until one is attached. Labels control which test tier runs and whether a
new container image is built.
| Label | What runs | Container |
|---|---|---|
CI:docs |
Doc tests only | Reuses main container |
CI:Lfast |
Fast test subset | Reuses main container |
CI:L0 |
Unit tests + docs + lint | Builds new image |
CI:L1 |
L0 + functional tests | Builds new image |
CI:L2 |
L1 + convergence tests | Builds new image |
Skip CICD |
Nothing (skips all tests) | — |
Default on merge group / push to main: L1.
Which label to attach when opening a PR:
| Changed paths / nature of change | Label |
|---|---|
Docs only (docs/, *.md, docstrings) |
CI:docs |
| Trivial fix, no logic change | CI:Lfast |
| New code, bug fix, refactor | CI:L0 |
| Changes that could affect model behaviour | CI:L1 |
| Changes that could affect convergence | CI:L2 |
CI Failure Investigation
# List recent workflow runs for the PR
gh run list --repo NVIDIA-NeMo/RL --branch "pull-request/<pr-number>"
# View failing run summary
gh run view <run-id> --repo NVIDIA-NeMo/RL
# Stream failing job output
gh run view <run-id> --repo NVIDIA-NeMo/RL --log-failed
Common failure patterns:
| Symptom | Likely cause | Fix |
|---|---|---|
| CI never starts | No trust signal or copy-pr-bot not triggered | Post /ok to test <sha> |
semantic-pull-request fails |
PR title doesn't follow Conventional Commits | Fix PR title; see contributing skill |
| Linting fails | Style violation | Run uv run ruff check --fix . && uv run ruff format . |
| Unit test failure | Code regression or missing dependency | Reproduce locally; see testing skill |