CI/CD
Commit and PR Workflow
- Never commit directly to
main — always create a feature branch.
- Always sign commits:
git commit -s -m "message".
- PR title format: Conventional Commits —
<type>(<scope>): <description>
(e.g., feat(model): add Qwen3 model bridge).
See @CONTRIBUTING.md for the full PR workflow, type/scope taxonomy, and DCO requirements.
How CI Is Triggered
The workflow is defined in @.github/workflows/cicd-main.yml and is triggered
on push — not on pull_request. This is intentional: a bot called
copy-pr-bot controls when CI runs.
Mechanism:
- When a PR is opened,
copy-pr-bot watches for a trust signal.
- Trust is established in one of two ways:
- All commits on the PR branch are GPG-signed by a verified NVIDIA contributor → bot triggers automatically.
- An NVIDIAN posts
/ok to test <commit-sha> as a PR comment → bot triggers manually for that SHA.
- Once trusted,
copy-pr-bot copies the PR's code into the remote branch
pull-request/<number> and pushes it.
- That push fires the workflow's
push trigger on refs/heads/pull-request/<number>,
launching CI.
Consequences:
- CI never runs on untrusted pushes — external contributors always need
/ok to test.
- The running workflow branch is
pull-request/<number>, not the author's feature branch.
- Pushing a new commit to a PR does not automatically re-trigger CI unless the
commit is signed or
/ok to test <new-sha> is posted.
- Concurrent runs for the same PR are cancelled automatically (concurrency group per PR number).
Pipeline Structure
pre-flight
└── lint-check
└── cicd-wait-in-queue # queues workflows to avoid runner interleaving across PRs
└── cicd-container-build
├── unit-tests-core
├── unit-tests-diffusion
└── functional-tests (L0 always; L1 with needs-more-tests label; L2 on schedule or full-test-suite label)
- Slack notifications are sent on completion for scheduled and nightly runs.
For functional test tier semantics and job-to-directory mapping, see the testing skill.
CI Failure Investigation
Locating the PR from a CI Branch
# Extract PR number from branch name (e.g. pull-request/1234)
PR_NUMBER=$(git rev-parse --abbrev-ref HEAD | grep -oP '(?<=pull-request/)\d+')
gh pr view "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge
gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge --name-only
gh pr checks "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge
Investigating a Failing Job
- Get the PR number from the branch name (see above).
- Review the changeset:
gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge
- Identify the failing job from
gh pr checks output.
- Fetch job logs:
gh run list --repo NVIDIA-NeMo/Megatron-Bridge --branch "pull-request/$PR_NUMBER"
gh run view <run_id> --repo NVIDIA-NeMo/Megatron-Bridge --log-failed > run.log
- Scan logs in chunks — log files can exceed 10,000 lines, never load them whole:
wc -l run.log
tail -200 run.log # start from the end
sed -n '1,200p' run.log # or scan forward in 200-line chunks
- Cross-reference the changeset against the failing step.
Hugging Face Model Access In CI
Assume CI functional-test containers run with Hugging Face models offline
(HF_HUB_OFFLINE=1) and a pre-populated HF_HOME. When reproducing or
fixing CI failures involving HF models, mirror this locally by setting
HF_HUB_OFFLINE=1 after warming the cache. Test fixtures must not depend on
live Hub API calls such as list_repo_files() or uncached downloads during CI.
For trust_remote_code=True toy checkpoints, copy custom Python modules from
the already loaded local/cache source files or a local snapshot, not by listing
the remote repo at test time.
Common Failure Patterns
| Symptom |
Likely Cause |
Action |
| CI never started on a PR |
Commits not GPG-signed and no /ok to test comment |
Post /ok to test <full-sha> on the PR |
| Lint job fails |
ruff or pre-commit violation |
Run ruff check --fix + ruff format locally |
| Container build fails |
Dependency conflict or stale uv.lock |
Re-run uv lock inside Docker and commit updated lock |
| Unit tests fail |
Code regression or missing import |
Run failing test locally; check the PR diff |
| Functional test (L0) fails |
Integration breakage |
Check GPU runner logs; reproduce with L0_Launch_*.sh |
HF model fixture passes locally but fails in CI with OfflineModeIsEnabled |
Test made a live Hugging Face Hub API/download call; CI has HF_HUB_OFFLINE=1 |
Warm local cache, reproduce with HF_HUB_OFFLINE=1, and change the fixture to use cached/local artifacts only |
cicd-wait-in-queue running long |
Many PRs queued; automation serializes runners to avoid interleaving |
Wait; or check queue depth in the Actions tab |
| MCore submodule mismatch |
Pinned commit out of sync |
Update 3rdparty/Megatron-LM submodule and re-lock |
| Stale checkpoint auto-resume |
nemo_experiments/ from a previous run exists |
rm -rf nemo_experiments before starting fresh |
| Port collision on Slurm (EADDRINUSE) |
ntasks-per-node=8 with torchrun |
Drop torchrun; use ntasks-per-node=8 with uv run python script.py |
1---2name: cicd3description: CI/CD reference for Megatron Bridge — pipeline structure, commit and PR workflow, CI failure investigation, and common failure patterns.4when_to_use: Investigating a CI failure, understanding the pipeline structure, writing a commit or PR, triggering CI, 'CI is red', 'how do I trigger CI', 'PR workflow', 'where are the logs', 'CI did not run', 'copy-pr-bot', '/ok to test'.5metadata:6 source: https://skillmd.com/skills/nvidia-nemo-megatron-bridge/cicd7---89# CI/CD1011## Commit and PR Workflow1213- **Never commit directly to `main`** — always create a feature branch.14- **Always sign commits**: `git commit -s -m "message"`.15- **PR title format**: [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) — `<type>(<scope>): <description>`16 (e.g., `feat(model): add Qwen3 model bridge`).17See @CONTRIBUTING.md for the full PR workflow, type/scope taxonomy, and DCO requirements.1819## How CI Is Triggered2021The workflow is defined in @.github/workflows/cicd-main.yml and is triggered22on `push` — **not** on `pull_request`. This is intentional: a bot called23`copy-pr-bot` controls when CI runs.2425**Mechanism:**261. When a PR is opened, `copy-pr-bot` watches for a trust signal.272. Trust is established in one of two ways:28 - All commits on the PR branch are **GPG-signed** by a verified NVIDIA contributor → bot triggers automatically.29 - An NVIDIAN posts `/ok to test <commit-sha>` as a PR comment → bot triggers manually for that SHA.303. Once trusted, `copy-pr-bot` copies the PR's code into the remote branch31 `pull-request/<number>` and pushes it.324. That push fires the workflow's `push` trigger on `refs/heads/pull-request/<number>`,33 launching CI.3435**Consequences:**36- CI never runs on untrusted pushes — external contributors always need `/ok to test`.37- The running workflow branch is `pull-request/<number>`, not the author's feature branch.38- Pushing a new commit to a PR does **not** automatically re-trigger CI unless the39 commit is signed or `/ok to test <new-sha>` is posted.40- Concurrent runs for the same PR are cancelled automatically (concurrency group per PR number).4142## Pipeline Structure4344```45pre-flight46 └── lint-check47 └── cicd-wait-in-queue # queues workflows to avoid runner interleaving across PRs48 └── cicd-container-build49 ├── unit-tests-core50 ├── unit-tests-diffusion51 └── functional-tests (L0 always; L1 with needs-more-tests label; L2 on schedule or full-test-suite label)52```5354- Slack notifications are sent on completion for scheduled and nightly runs.5556For functional test tier semantics and job-to-directory mapping, see the `testing` skill.5758## CI Failure Investigation5960### Locating the PR from a CI Branch6162```bash63# Extract PR number from branch name (e.g. pull-request/1234)64PR_NUMBER=$(git rev-parse --abbrev-ref HEAD | grep -oP '(?<=pull-request/)\d+')6566gh pr view "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge67gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge --name-only68gh pr checks "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge69```7071### Investigating a Failing Job72731. **Get the PR number** from the branch name (see above).742. **Review the changeset**:75 ```bash76 gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge77 ```783. **Identify the failing job** from `gh pr checks` output.794. **Fetch job logs**:80 ```bash81 gh run list --repo NVIDIA-NeMo/Megatron-Bridge --branch "pull-request/$PR_NUMBER"82 gh run view <run_id> --repo NVIDIA-NeMo/Megatron-Bridge --log-failed > run.log83 ```845. **Scan logs in chunks** — log files can exceed 10,000 lines, never load them whole:85 ```bash86 wc -l run.log87 tail -200 run.log # start from the end88 sed -n '1,200p' run.log # or scan forward in 200-line chunks89 ```906. **Cross-reference the changeset** against the failing step.9192### Hugging Face Model Access In CI9394Assume CI functional-test containers run with Hugging Face models offline95(`HF_HUB_OFFLINE=1`) and a pre-populated `HF_HOME`. When reproducing or96fixing CI failures involving HF models, mirror this locally by setting97`HF_HUB_OFFLINE=1` after warming the cache. Test fixtures must not depend on98live Hub API calls such as `list_repo_files()` or uncached downloads during CI.99For `trust_remote_code=True` toy checkpoints, copy custom Python modules from100the already loaded local/cache source files or a local snapshot, not by listing101the remote repo at test time.102103## Common Failure Patterns104105| Symptom | Likely Cause | Action |106|---|---|---|107| CI never started on a PR | Commits not GPG-signed and no `/ok to test` comment | Post `/ok to test <full-sha>` on the PR |108| Lint job fails | `ruff` or `pre-commit` violation | Run `ruff check --fix` + `ruff format` locally |109| Container build fails | Dependency conflict or stale `uv.lock` | Re-run `uv lock` inside Docker and commit updated lock |110| Unit tests fail | Code regression or missing import | Run failing test locally; check the PR diff |111| Functional test (L0) fails | Integration breakage | Check GPU runner logs; reproduce with `L0_Launch_*.sh` |112| HF model fixture passes locally but fails in CI with `OfflineModeIsEnabled` | Test made a live Hugging Face Hub API/download call; CI has `HF_HUB_OFFLINE=1` | Warm local cache, reproduce with `HF_HUB_OFFLINE=1`, and change the fixture to use cached/local artifacts only |113| `cicd-wait-in-queue` running long | Many PRs queued; automation serializes runners to avoid interleaving | Wait; or check queue depth in the Actions tab |114| MCore submodule mismatch | Pinned commit out of sync | Update `3rdparty/Megatron-LM` submodule and re-lock |115| Stale checkpoint auto-resume | `nemo_experiments/` from a previous run exists | `rm -rf nemo_experiments` before starting fresh |116| Port collision on Slurm (EADDRINUSE) | `ntasks-per-node=8` with `torchrun` | Drop torchrun; use `ntasks-per-node=8` with `uv run python script.py` |