Megatron LM

CI/CD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request/BRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.

Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.

Guides testing Megatron-LM: test layout, recipe YAML, adding and running unit/functional tests, golden values, marker filters, and CI parity.

Split a large pull request into multiple smaller PRs to reduce the number of required CODEOWNERS reviewer groups.

Research and draft a response to a GitHub issue or question from an external contributor.

Investigate a failing GitHub Actions job, extract the root cause, and file a well-structured bug issue against NVIDIA/Megatron-LM.

Launch distributed Megatron-LM training jobs on a SLURM cluster with a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules, container conventions, monitoring, and per-rank failure diagnosis.

Refresh golden values from a GitHub Actions workflow run (failing-only or all jobs), calculate signed per-model percentage changes, and produce a PR-ready summary. Use when the user asks to update goldens for a CI run, refresh golden values from a workflow ID, or generate a golden-value diff summary for a PR description.

Bump the NVIDIA PyTorch base image (`nvcr.io/nvidia/pytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker/.ngc_version.dev` and GitLab CI in `.gitlab/stages/01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs

Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.

Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script.

Lint and format Python code for Megatron-LM using ruff, black, isort, pylint, and mypy, with commands for autoformatting and import ordering.

Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.