# Create Model Verification Card

> Create or update concise, agent-readable Megatron Bridge model verification cards. Use when adding a model support card, auditing cross-model convergence comparability or verification coverage, recording conversion, deterministic inference, training, checkpoint resume, post-SFT export, performance, or weak-scaling results, or preparing a model-support PR. Enforce the required core inventory, convergence-versus-performance contracts, optional performance items, public Slurm launcher commands, training metrics, important-feature allowlist, and a strict privacy boundary that excludes private runtime wiring, internal paths, credentials, and job metadata.

- Skill: `nvidia-nemo-megatron-bridge/create-model-verification-card` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add nvidia-nemo-megatron-bridge/create-model-verification-card`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-nemo-megatron-bridge/create-model-verification-card/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: NVIDIA-NeMo (https://skillmd.com/u/nvidia-nemo-megatron-bridge)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/nvidia-nemo-megatron-bridge/create-model-verification-card

---


# Create Model Verification Card

Create `examples/model_verification_cards/<model-slug>/card.yaml` from public
model facts and verified commands. Keep the card small enough for an agent to
scan without interpreting logs or reconstructing the execution environment.

## Use the repository resources

Treat verification scripts, validators, and launchers as shared infrastructure. Do not modify them merely to make
one model card pass. Any such change requires a clear, documented, reusable reason: identify the existing behavior
that is insufficient, the affected public workflows or models, why an existing maintained path cannot be used, and
add focused backward-compatible tests. Record the justification in the PR description or commit. If the need is
model-specific or the reason is not clear, leave the affected verification item unverified instead of adding a
card-only workaround.

- Validate the result with [scripts/validate_card.py](scripts/validate_card.py).
- Verify deterministic HF output with
  [scripts/verify_hf_inference.py](scripts/verify_hf_inference.py).
- Use the inventory and field rules below as the format contract. Do not infer
  model-specific settings from another family or variant.

Do not add a README, evidence blobs, log excerpts, runtime setup, or scheduler
metadata to the skill or card.

## Workflow

### 1. Pull facts before drafting

Read the model implementation, public HF config, conversion bridge, recipes,
tests, examples, and the exact revision being verified. Determine which modes
exist; do not infer support from a family name or another model size.

Record the public HF model name in commands, its immutable revision in
`model.hf_revision`, and the minimum supported Transformers version in
`model.min_transformers_version`. Do not introduce an HF snapshot path.

Record only two execution-environment facts: the public base container
identifier and the exact Bridge commit used for verification. Put them in
`verification_environment.base_container` and
`verification_environment.bridge_commit`. If the run mounted a checkout over
the container, record the mounted checkout commit. Never substitute a private
image path for the public base container identifier.

The top-level Bridge commit is the default for every item leaf. If one verified
item was run from a different clean checkout, put that exact 40-hex commit in
the leaf's optional `bridge_commit` field. Omit the field when it would repeat
the top-level value, and never use a commit field to disguise uncommitted
runtime changes. Items that are not verified must not carry a commit override.

### 2. Create the core inventory and add performance when available

Include these twelve required items, even when their status is `unsupported` or
`not_applicable`:

1. CPU HF-to-Megatron conversion
2. GPU HF-to-Megatron conversion
3. CPU Megatron-to-HF conversion
4. GPU Megatron-to-HF conversion
5. Manual HF/Megatron forward-pass correlation
6. Deterministic Megatron inference
7. Pretraining
8. SFT
9. SFT checkpoint export and deterministic HF inference
10. Long-context SFT
11. PEFT
12. Checkpoint resume

Add `pretrain_performance` as a thirteenth item only when the exact model
variant has a canonical public performance recipe. If no such recipe is
exported, omit the item instead of adding an unverified placeholder. Once a
canonical recipe exists, keep the item in the card even if its run is still
unverified.

Add `pretrain_fsdp` as an optional hardware-scoped item when a first-class
Megatron FSDP performance recipe exists for the exact model variant. Record
the FSDP result under `pretrain_fsdp.<hardware>`. Keep it separate from
checkpoint-resume and tuned non-FSDP `pretrain_performance` results. Do not
embed a baseline, control, computed delta, or relative speedup under another
item; agents can derive valid comparisons from the standalone commands,
resolved recipes, and raw metrics. The FSDP item does not claim checkpoint
save/load unless a separate functional item records that evidence.

When the same hardware has multiple first-class FSDP runs with different
precisions or convergence contracts, use one aggregate hardware container and
key its standalone leaves by precision:

```yaml
pretrain_fsdp:
  GB200:
    status: verified
    variants:
      bf16:
        status: verified
        precision: bf16
        # complete standalone leaf
      fp8_mx:
        status: verified
        precision: fp8_mx
        # complete standalone leaf
```

The container status is `verified` only when every precision variant is
verified; otherwise it is `unverified`. Keep the ordinary direct hardware-leaf
shape when there is only one FSDP run. A precision variant repeats its
precision as a scalar so agents do not have to infer workload facts from a
mapping key.

Add `pretrain_weak_scaling` only when at least two completed runs use the same
canonical performance recipe, precision, sequence length, maximum step count,
and GPUs per node while global batch size grows in direct proportion to total
GPU count. Omit it from models without measured scaling data. Put it last in
`items`, key it by public hardware, and keep shared provenance on the hardware
leaf. Each ordered point records only its GPU count, GBS, public command, and
the standard five training metrics:

```yaml
pretrain_weak_scaling:
  GB300:
    status: verified
    precision: fp8_mx
    bridge_commit: 0123456789abcdef0123456789abcdef01234567
    last_verified: 2026-08-15
    points:
      - num_gpus: 8
        global_batch_size: 512
        command: >
          ./scripts/training/train.sh --nodes 2 --gpus-per-node 4
          --recipe example_pretrain_config --mode pretrain --max_steps 50
          --seq_length 4096 --global_batch_size 512
        metrics: # standard five training metrics
    expected_result: All points complete with finite metrics and no skipped or NaN iterations.
```

The weak-scaling item is the sole exception that may set global batch size in
the public command. It must not override micro batch size. Record raw point
metrics only; derive aggregate throughput and scaling efficiency when reading
the card rather than storing comparison payloads.

A concrete `pretrain_performance.<hardware>` leaf means a tuned canonical
performance recipe exists for that hardware. Its item status states whether
the card's benchmark run has been verified; an `unverified` leaf still records
the existence of the recipe. If the card has no concrete performance leaf,
start `summary` with this exact disclaimer before the functional-support
summary:

```text
Performance disclaimer: this card does not record a canonical pretrain performance result; reported timing and throughput metrics are functional verification observations, not standalone optimized performance results.
```

This wording describes the evidence recorded by the card, not whether the
model has ever been tuned elsewhere. The validator continues to accept the
older untuned-model wording on existing cards for backward compatibility, but
new and updated cards must use the wording above.

Omit `pretrain_performance` entirely when no canonical recipe exists; do not
add an `all` terminal placeholder. Routine timing and throughput metrics in
functional pretrain, SFT, PEFT, long-context, and resume items remain sanity
observations; they do not become optimized results merely because a separate
performance recipe exists. When a tuned recipe exists, scope the summary to
the exact `pretrain_performance.<hardware>` leaf.

Keep conversion, manual-forward, and base-inference items as direct item
records. Key every training item by its public hardware target, and key the
dependent SFT export/inference item the same way:

```yaml
items:
  pretrain:
    H100:
      status: verified
      precision: bf16
      enabled_features: {}
      command: ...
      last_verified: 2026-07-21
      metrics: ...
      expected_result: ...
```

The hardware-scoped names are `pretrain`, `sft`, `sft_export_inference`,
`sft_long_context`, `peft`, `checkpoint_resume`, and optional
`pretrain_performance`, `pretrain_fsdp`, and `pretrain_weak_scaling`. Use
canonical public accelerator identifiers such as `H100`, `B200`, or `GB200`,
never a private cluster name.
The validator's public-hardware allowlist is authoritative and must be updated
when a new accelerator target is introduced. The hardware key replaces the old
`gpu_type` field. Each hardware leaf is independent and must carry its own
status plus the command or commands, date, metrics, features, and optional
commit override that apply to that item. Dependencies resolve within the same
hardware key: `checkpoint_resume.H100` consumes `pretrain.H100`, and
`sft_export_inference.H100` consumes `sft.H100`. Never fall back across hardware
targets. Use the reserved key `all` only as the sole leaf for a model-wide
`unsupported` or `not_applicable` limitation. A terminal dependency leaf still
names its logical dependency but does not require a matching `all` or
concrete-hardware dependency leaf.

Use only `unverified`, `verified`, `unsupported`, or `not_applicable`. Do not
add `smoke` or an evidence field.

#### Add a compact verification index

Every card must put a top-level `verification_index` immediately after
`summary`. This is a concise directory of the detailed `items`, not a second
source of evidence. The validator must reject an index that drifts from the
item records.

Use `model_level` for the six direct items: the four conversion directions,
`manual_forward_pass`, and `inference`. Use `training` for the six functional
hardware-scoped items: `pretrain`, `sft`, `sft_export_inference`,
`sft_long_context`, `peft`, and `checkpoint_resume`. Keep the optional
`pretrain_performance` item separate under `performance`; omit `performance`
when the card has no canonical performance recipe. Keep optional
`pretrain_fsdp` leaves separate under `fsdp`; omit `fsdp` when the card has no
FSDP recipe. Mirror optional `pretrain_weak_scaling` leaves under
`weak_scaling`; omit that index when the card has no measured scaling item.

Group item names under the same four status names used by the detailed items.
For an explicitly indexed hardware target with no corresponding item leaf,
summarize the missing verification as `unverified`; do not add empty item
leaves or placeholder commands merely to populate the index. Omit empty status
buckets and always list every item name explicitly, even when every item in the
scope has the same status. Never use the scalar `all` in `verification_index`:
an explicit inventory makes it clear which existing items were assessed when
new items are added later.

For example, a card with partial H100 functional-training coverage and no
GB200 functional-training verification uses:

```yaml
verification_index:
  model_level:
    verified:
      - hf_to_megatron_cpu
      - hf_to_megatron_gpu
      - megatron_to_hf_cpu
      - megatron_to_hf_gpu
      - manual_forward_pass
      - inference
  training:
    H100:
      verified: [sft, sft_export_inference, sft_long_context, peft]
      unverified: [pretrain, checkpoint_resume]
    GB200:
      unverified: [pretrain, sft, sft_export_inference, sft_long_context, peft, checkpoint_resume]
```

When a canonical performance recipe exists, mirror only its concrete leaves:

```yaml
  performance:
    H100: verified
```

When an FSDP recipe exists, mirror only its concrete leaves:

```yaml
  fsdp:
    GB200: verified
```

When weak-scaling measurements exist, mirror only their concrete leaves:

```yaml
  weak_scaling:
    GB300: verified
```

For a multi-variant FSDP hardware container, this scalar mirrors the aggregate
container status rather than duplicating the per-precision inventory.

Do not add prose comparisons, control payloads, computed deltas, or relative
speedups to performance leaves. Agents can compare standalone runs
automatically after resolving their recipes. The only exception is a concise
warning that names two superficially similar runs that must **not** be compared
and the exact convergence-contract differences that make them incompatible.

The index may declare an allowlisted public hardware target such as `GB200`
before detailed evidence exists. It must also include every concrete hardware
target present in the detailed items. Do not use a private cluster name or
invent a separate `not_tested` status; `unverified` covers both pending and
not-yet-run verification.

For `unsupported` and `not_applicable`, leave `command` or `commands`, date,
precision, and metrics null, then state the public product limitation in
`expected_result`.

Put a scalar `precision` on every direct item or hardware leaf. It describes
the workload that was actually verified, not every precision the model might
support:

- for conversion, record the imported or exported weight precision;
- for forward pass and inference, record the compute precision;
- for training, record the recipe's mixed-precision mode.

Use `bf16` for BF16. Training items may instead use `fp8_mx` for MXFP8 or
`nvfp4` for NVFP4. Keep MXFP8 and NVFP4 training-only, and do not list either
until that exact item has completed in that mode.

The outer hardware container of a multi-variant `pretrain_fsdp` item is the
only exception: each variant owns the scalar precision and the container owns
only aggregate `status` plus `variants`.

### 3. Use the public Slurm launchers

Assume the caller supplies the account, partition, concrete runtime image,
credentials, and storage mappings outside the card. The top-level
`verification_environment.base_container` is provenance only; it is not
launcher configuration. Record the portable public launcher and the
model-verification workload:

- use `scripts/conversion/convert.sh --executor slurm` for CPU and GPU
  conversion, with portable node and GPU counts;
- use `scripts/training/train.sh --nodes ... --gpus-per-node ...` for every
  training item;
- use `scripts/inference/infer.sh --nodes ... --gpus-per-node ... --task ...`
  for Megatron inference and manual model comparison; select
  `text-generation`, `legacy-full-prefix-generation`, `vlm-generation`, or
  `model-comparison` explicitly. The legacy full-prefix task is a slow,
  non-optimized compatibility path and requires `--legacy-full-prefix`;
- invoke public shell launchers directly from the card. Do not create Slurm
  jobs by calling their Python setup modules, and do not wrap the launchers in
  `uv run`, `python`, `srun`, or `sbatch`; the shell entry point owns its Python
  environment and scheduler setup;
- use `uv run python ...` only for a local artifact verifier that has no public
  shell wrapper, such as deterministic Hugging Face inference after export. It
  must not be used as a substitute for an available shell launcher or to create
  a cluster job;
- use short, ignored repository-relative logical paths under `work/...`;
  prefer aliases such as `work/data/<dataset>` and `work/cache/<model>` over
  reproducing a physical storage hierarchy.

CPU conversion should omit `--gpus-per-node` when it is genuinely CPU-only.
If runtime construction demonstrably requires CUDA even though model weights
remain on CPU, request exactly one shared runtime GPU and explain the exception
in the item's result. Never request a full GPU node merely to satisfy a
launcher or monitor.

For example, a portable multi-node VLM verification starts with:

```bash
./scripts/inference/infer.sh --nodes 4 --gpus-per-node 8 \
  --task vlm-generation \
  --hf_model_path <org>/<model> \
  --megatron_model_path work/model-verification/<model>/iter_0000000 \
  --image_path docs/images/tp1.png \
  --prompt "Describe this image." \
  --max_new_tokens 64
```

The public launchers may read their required generic Slurm configuration from
the caller's environment. Do not include `srun`, `sbatch`, concrete account or
partition values, container image arguments, `--mount`, `--env`, shell exports,
environment-variable references, cluster-specific `--srun-arg` values, or
launcher overlays. That wiring is personal to the verifier and does not belong
in a model verification card.

Never record or reproduce:

- execution-environment names, hostnames, IPs, usernames, emails, or accounts;
- concrete partitions, reservations, node lists, private image locations,
  mount sources, or environment forwarding;
- host/shared-storage paths, home-directory paths, log locations, or job IDs;
- tokens, token-loading commands, private URLs, or private registry references;
- environment-specific launcher overlays.

Keep private run notes outside the tracked repository and public PR text. If a
private codename cannot be recognized generically, pass it to the validator via
`--deny-term` or an untracked file through `--denylist "$PRIVATE_DENYLIST"`.
The validator reports the match without printing the private term.

### 4. Freeze convergence, execution, and benchmark contracts

Before launching any training item, resolve the selected recipe and classify
its effective configuration into the three groups below. Do this from the
built `ConfigContainer`, not only from command-line overrides. Record the
resolved convergence and execution field/value fingerprints in the internal
per-model record; keep the public card concise.

The **convergence contract** contains settings that change the training
objective, examples seen at an optimizer boundary, or numerical update rule:

- starting checkpoint and trainable parameter set;
- dataset identity/revision, split or bounded selection, sample order, seeds,
  tokenizer/chat template, truncation, masking, and packing semantics;
- sequence length, global batch size, global tokens per optimizer step, total
  optimizer steps, and total token budget;
- objective and loss settings, including label masking, MoE auxiliary/router
  losses, token dropping/capacity, natural versus forced routing, and loss
  normalization;
- optimizer family, peak/minimum learning rate, schedule shape, warmup and
  decay horizon, betas, epsilon, weight decay, gradient clipping, and dropout;
- model/gradient/optimizer-state precision and loss-scaling behavior;
- for PEFT, adapter type, targets, rank, alpha, dropout, and which base weights
  are frozen.

The **execution/performance contract** maps the frozen training semantics to
hardware. It may vary across models or be tuned for throughput:

- node/GPU count and TP, PP, VP, CP, EP, ETP, DP, and sequence parallelism;
- activation recompute, activation/optimizer offload, distributed optimizer or
  FSDP sharding, checkpoint I/O strategy, and garbage-collection policy;
- communication overlap, fused kernels, Transformer Engine implementation,
  attention backend, CUDA graphs, and compilation;
- MoE transport/dispatcher backend such as all-to-all, DeepEP, or HybridEP,
  provided routing, capacity, token dropping, and auxiliary losses are
  unchanged.

Treat micro batch size and gradient-accumulation layout as execution
fingerprints. They may be tuned while global batch size, global batch
membership/order, loss normalization, optimizer-step boundaries, and total
token budget remain intact. Because they can change accumulation order,
dropout RNG, MoE token grouping, and auxiliary-loss reduction, require fresh
loss sentinels for every layout and do not claim step-by-step numerical parity.

Performance settings are **intended** to preserve training semantics, not
guaranteed to be bitwise neutral. Parallel reductions, fusions, recompute, and
dispatcher implementations can change floating-point order. Every paired run
used to claim that a performance-related feature preserves convergence must
start from the same weights and use the same clean Bridge commit, dataset and
revision, sample order, tokenizer, sequence length, global batch size, token
budget, optimizer steps, seeds, objective, routing, precision, optimizer, and
learning-rate schedule. Change only the feature under test and unavoidable
execution settings. Keep micro batch size and gradient accumulation unchanged
when possible; if either changes, require the same loss check but do not claim
step-by-step numerical identity. If a verified run deliberately changes any
convergence field, make the exact deviation explicit in its command or
`expected_result`.

Compare LM and auxiliary-loss values at every shared optimizer step. Each
candidate value must satisfy
`abs(candidate - reference) <= 1e-6 + 0.01 * abs(reference)`. Require finite
losses, zero skipped or NaN iterations, and the same qualitative loss trend.
If any convergence field differs or any loss falls outside this bound, do not
claim that the feature is convergence-neutral. Treat the result as standalone
performance evidence, investigate the discrepancy, and explicitly identify
the pair as not comparable only when the card would otherwise invite a false
comparison. Anything that changes arithmetic precision, forced router
balancing, token dropping, packing, or effective batch construction is a
convergence change, even when introduced to improve speed.

The **benchmark-only configuration** may deliberately change semantics to find
an upper throughput bound. It includes mock data, forced MoE load balancing,
changed batch/LR or timing-only schedules, and disabled NaN/large-gradient,
evaluation, or checkpoint checks. These settings may be valid for a canonical
`pretrain_performance` item, but their losses and checkpoints are never
convergence evidence.

Use `qwen3_30b_a3b_convergence_v2` as the named default cross-model bounded
convergence cohort. It retains the optimizer, data, objective, and token-budget
contract derived from the resolved Qwen3-30B-A3B H100 recipes while changing
offline-packed SFT and PEFT from 2K/GBS32 to 8K/GBS8. The name identifies
target settings, not evidence status.
Do not call a workload cohort-verified until its recipe owns this contract and
a clean-commit run passes the applicable gates below. Its 100 optimizer steps
test finite loss, short-horizon loss trend, checkpoint reload, and direct
resume; they do not establish full training convergence.

Historical `qwen3_30b_a3b_convergence_v1` evidence used 2K/GBS32 for SFT and
PEFT. Keep an existing verified card on that exact command, recipe commit, and
metrics until the model is rerun. Never relabel 2K evidence as v2 or rewrite its
command to 8K without fresh training. When refreshing a model, update the
recipe first, write packed data to a fresh output root, rerun the training
gates, and rerun post-SFT export/inference.

Freeze this optimizer fingerprint for all three workloads:

| Field | Value |
| --- | --- |
| Optimizer | Megatron distributed fused Adam |
| Betas / epsilon | `(0.9, 0.95)` / `1e-8` |
| Effective weight decay | `0.033`, constant |
| Gradient clipping | `1.0` |
| LR schedule | Cosine, starting from zero |
| Model / compute precision | BF16 |
| Optimizer master parameters, main gradients, moments | FP32 |

Record the effective weight decay applied to optimizer parameter groups, not
only the nominal optimizer-config value. The Qwen anchor has a nominal
`optimizer.weight_decay=0.1`, but its constant scheduler applies `0.033`; use
`0.033` when reproducing this contract. Treat any other effective value
as a convergence-contract change.

Freeze the following workload profiles:

| Field | Pretrain | Full SFT | PEFT |
| --- | --- | --- | --- |
| Start / trainable set | MoE: exact immutable imported Megatron checkpoint, full model; dense: random initialization, no checkpoint load, full model | Exact immutable HF checkpoint revision, full model | Same immutable HF revision, frozen base model; LoRA on model-native attention Q/K/V and output projections, rank 8, alpha 16, dropout 0 |
| Data | Same bounded raw RP2 selection, revision, sample order, and seeds | Tulu 3 `train[:10000]`; same revision, order, chat template, label mask, truncation, and offline packing | Same as full SFT |
| Sequence / GBS | `4096 / 1024` | `8192 / 8` | `8192 / 8` |
| Reference MBS | `1` | `1` | `1` |
| Offline packing alignment | Not applicable | Derive from resolved CP/TP/SP topology and pin explicitly | Same rule |
| Token slots | `4,194,304` per step; `419,430,400` total | `65,536` per step; `6,553,600` total | `65,536` per step; `6,553,600` total |
| Peak / minimum LR | `3e-4 / 3e-5` | `5e-6 / 0` | `1e-4 / 0` |
| Horizon | 100 steps, 40 warmup steps, cosine decay through step 100, saves at steps 50 and 100 | 100 steps, 10 warmup steps, cosine decay through step 100, final checkpoint at step 100 | 100 steps, 10 warmup steps, cosine decay through step 100, final adapter checkpoint at step 100 |
| RNG | Model and dataset seed `1234` | Model RNG seed `5678`; data-order and packing seed `1234` | Model RNG seed `5678`; data-order and packing seed `1234` |
| Gradient path | BF16 gradient reduction; precision-aware optimizer enabled | FP32 gradient reduction; precision-aware optimizer disabled | FP32 gradient reduction; precision-aware optimizer disabled |

For MoE pretraining, keep pretrained-checkpoint selection out of the reusable
recipe. The resolved recipe must leave `checkpoint.pretrained_checkpoint`
unset; the model-verification-card command must pass exactly one
`--pretrained_checkpoint` pointing to the card's immutable imported Megatron
checkpoint. This is a weight-only warm start, not a full-state resume, so the
uninterrupted reference command must still keep `checkpoint.load=null` and must
not use `--load_dir`. Starting from trained router weights makes natural expert
traffic, load imbalance, memory use, and reported TFLOP/s more representative
than random router initialization.

Apply the same launch-time checkpoint rule to any MoE
`pretrain_performance` item that claims natural-routing throughput. A
benchmark-only forced-balance recipe may omit the checkpoint because its router
traffic is synthetic by construction; state that distinction explicitly.
Never add a checkpoint path to a library or performance recipe merely to make a
card pass. Historical verified random-initialization evidence may remain on its
exact recorded command, but do not rewrite it as checkpoint-backed evidence;
adopt this contract when that item is rerun.

Derive `pad_seq_to_mult` for both SFT and PEFT from the resolved execution
topology:

```text
cp_multiple = 2 * CP if CP > 1 else 1
sp_multiple = CP * TP if sequence parallelism is enabled and TP > 1 else 1
pad_seq_to_mult = lcm(cp_multiple, sp_multiple)
```

The two workloads use the same rule; there is no intrinsic SFT-versus-PEFT
alignment difference. Offline packing does not finalize this value
automatically, so set the derived integer explicitly in the recipe and card
command. Because changing it changes padding and pack membership, use a fresh
packing output, record the resolved value, packing-manifest hash, and actual
supervised-token count, and require fresh loss sentinels after a topology
change.

Use 8K as the default offline-pack target only when the exact model supports at
least 8192 tokens, the recipe supports offline packing, and the resolved
topology fits one MBS1 pack per data-parallel rank. Set `model.seq_length`,
`dataset.seq_length`, and `packed_sequence_size` to 8192 together. A
model-family packing opt-out, MTP incompatibility, context limit, or
demonstrated memory limit is a cohort exception, not a reason to silently fall
back.

Treat fixed-width pack padding as an execution requirement, not part of the
cross-model convergence profile. Set `pad_to_max_length=true` when the selected
dispatcher or kernel requires a fixed token width; for example, the verified
Moonlight HybridEP path requires the pack width to be divisible by its
128-token combine chunk. CUDA graphs also require fixed token width and
`pad_cu_seqlens=true` plus packing metadata. Otherwise do not require
fixed-width padding universally. Any padded tail must retain a zero loss mask.

The equal-token rule is:

```text
token_slots_per_step = packed_sequence_size * global_batch_size
```

Moving from 2K/GBS32 to 8K/GBS8 preserves 65,536 token slots per optimizer step
and can reduce gradient accumulation while presenting more source sequences in
each physical MBS1 pack. It still changes truncation and packing membership, so
it is a convergence-contract migration that requires fresh loss sentinels.
Because offline packing requires MBS1, require `GBS % DP == 0` and `GBS >= DP`.
For the v2 GBS8 target, select a model-appropriate topology with DP in
`{1, 2, 4, 8}`; do not reuse a DP16 layout.

The public training runner applies `--dataset` after constructing the model
recipe and replaces the recipe's dataset object with the selected preset.
Consequently, a card command that uses `--dataset tulu3` must explicitly pin
the dataset revision, split, data-order/packing seed, offline-packing enablement,
and the derived `+dataset.offline_packing_specs.pad_seq_to_mult`; recipe-level
dataset defaults alone do not freeze the resolved CLI workload. When the
execution requires fixed-width packs, also pin
`dataset.dataset_kwargs={pad_to_max_length:true}` and, for CUDA graphs,
`+dataset.offline_packing_specs.pad_cu_seqlens=true`. Use a fresh
`dataset.hf_output_root` (or force a deliberate rewrite) whenever sequence
length, packing width, alignment, or any of these fields changes, and audit the
final `ConfigContainer` after all overrides and runtime synchronization.

Before launching training, create the parent directory named by
`logger.save_config_filepath` and require the post-setup file to persist.
Treat only that saved, post-synchronization `ConfigContainer` as runtime-config
evidence. YAML printed by the recipe runner before setup is launch-time
configuration and may differ after model finalization; never relabel it as
resolved runtime evidence or combine it with another run's logs. If the file
does not persist, fix the output path and rerun the workload from a fresh root.

Treat the token counts above as token slots. For SFT and PEFT, also record the
actual supervised-token count after label masking; do not present padded or
masked token slots as supervised tokens.

Use these accumulation constraints for the reference executions:

| Workload | Topology constraint | DP | Gradient accumulation |
| --- | --- | ---: | ---: |
| Pretrain | 16 GPUs, TP1/PP1/CP1/EP16 | 16 | 64 |
| Full SFT | Model-appropriate topology with DP dividing GBS8 | 1, 2, 4, or 8 | `8 / DP` |
| PEFT | Model-appropriate topology with DP dividing GBS8 | 1, 2, 4, or 8 | `8 / DP` |

Topology, DP, and gradient accumulation are execution fingerprints rather than
convergence constraints. They may change while GBS8 remains fixed. Every new
execution layout must pass fresh loss sentinels, and its step-by-step values
are not strictly numerically comparable with another layout. The current
offline-packed SFT implementation requires MBS1; treat that as an
implementation limit, not a convergence rule.

For the Qwen anchor, record 128 experts, top-8 post-softmax-normalized routing,
auxiliary load-balancing loss coefficient `1e-3`, natural routing, and no
forced balancing or token dropping. These are model-native identity fields,
not universal cross-model overrides. Keep another model's native expert count,
top-k, router objective, auxiliary loss, capacity, and dropout values, record
them in its convergence fingerprint, and never alter them merely to imitate
Qwen. Require natural routing and prohibit benchmark-only forced balancing in
all convergence cohorts.

The Qwen PEFT anchor names its fused attention projections `linear_qkv` and
`linear_proj`. Preserve the same semantic adapter scope on architectures that
split Q, K, and V: list every model-native attention projection explicitly and
record the names as a model-specific fingerprint. For Moonlight MLA this is
`linear_q_proj`, `linear_kv_down_proj`, `linear_kv_up_proj`, and
`linear_proj`. Never retain a nonexistent fused-module name merely to make the
textual configuration look identical; verify that every declared target
actually matches modules before accepting PEFT evidence.

Use the same numerical value for global batch size within a comparison cohort.
If a model cannot use that value, change and validate its recipe separately,
record the exception, and treat the result as support verification rather than
an apples-to-apples convergence comparison. Compare progress at equal processed
token counts as well as equal optimizer steps. Different model architectures
and tokenizers make absolute cross-model loss values non-comparable; the shared
contract supports comparisons of stability and loss trend, not a ranking by
final loss.

Pin the same raw-document selection across models, then tokenize it with each
model's verified tokenizer unless a deliberately shared tokenizer is part of
the cohort. Do not assume that one indexed token-ID prefix represents the same
text under different tokenizers. The Qwen anchor's pinned tokenizer revision
may reproduce the Qwen run, but do not silently reuse its token IDs for another
model family. Compare cross-model stability and trend, never absolute loss.

Make every bounded convergence override explicit in the card command and apply
the same cohort values across models; never tune these values merely to improve
throughput. Library recipes own global and micro batch size. If a recipe batch
disagrees with the cohort contract, update and validate the recipe separately
instead of overriding it in the card. A canonical `perf_recipes` benchmark may
intentionally use mock data, forced balancing, or a different batch and is not
convergence evidence.

Keep long-context SFT outside `qwen3_30b_a3b_convergence_v2`. Sequence length,
CP, packing, batch construction, LR, and horizon define a separate convergence
cohort even when the starting checkpoint and dataset are shared.

### 5. Apply the verification gates

Mark an item `verified` only after the workload represented by its command has
completed with the recorded model, recipe, data, and checkpoint arguments and
the concrete expected result has been checked. A successful detached Slurm
submission is not completion; wait for the submitted workload and inspect its
result. Private executor configuration stays outside the card.

- **Conversion:** Test CPU and GPU import/export separately. Reload every
  output. Require exact keys, shapes, dtypes, and values when the conversion is
  expected to be lossless; otherwise state the numerical tolerance. Do not use
  `--detach` or a dry-run flag in a verified conversion command. Keep the card
  workload itself to import or export; do not run logit comparison or a
  roundtrip from a model's `conversion.sh`. Record numerical comparison as a
  separate inference workload.
- **Manual forward pass:** Compare Hugging Face and Megatron logits on the same
  prompt with `scripts/inference/infer.sh --task model-comparison`, which routes
  to `examples/conversion/compare_hf_and_megatron/compare.py`. Record whether
  the next token matches, the cosine similarity, and the maximum and mean
  absolute logit differences. For new evidence, pass `--hf-revision` with the
  exact `model.hf_revision` so the command itself is reproducibly pinned.
  Historical evidence verified before 2026-07-20, when the helper gained
  explicit revision pinning, may remain verified without a rerun when its
  clean-run provenance is tied to the card's immutable `model.hf_revision`;
  state this grandfathering explicitly in `expected_result`. Do not use the
  exception for unverified items or evidence dated 2026-07-20 and later.
  Mark the item verified when the next token matches and cosine similarity is
  at least 0.99 (cosine distance at most 1%). Numeric maximum and mean absolute
  differences are required diagnostic observations, but are report-only and
  must not guard the item status. This gate establishes functional logit
  correlation, not strict numerical equality. Keep this result separate from
  generation. Choose a prompt whose tokenized length is divisible by TP so the
  helper does not append padding before selecting the compared next-token
  position.
- **Megatron inference:** Launch through `scripts/inference/infer.sh` with the
  explicit `text-generation`, `legacy-full-prefix-generation`, or
  `vlm-generation` task. Use the slow, non-optimized legacy task only when
  cached inference is unsupported, and pass `--legacy-full-prefix`. Disable
  sampling and run one deterministic greedy generation with a maximum
  new-token bound. Allow natural end-of-sequence stopping, and record the
  actual generated-token count plus the literal completion including
  whitespace. A second replay may help diagnose nondeterminism, but it is not
  required verification evidence. Specify positive node and GPU counts and run
  synchronously; do not use `--detach` or a dry-run flag in a verified command.
- **Pretrain:** Use a bounded public dataset description and a stable schedule.
  For every MoE reference that measures natural-routing behavior, pass the
  immutable imported checkpoint with `--pretrained_checkpoint` in the card
  command while leaving the selected recipe's pretrained checkpoint unset.
  Do not use `--load_dir` for this weight-only initialization.
  Save a middle and final checkpoint when resume is in scope. For expensive
  workloads, a 100-step reference with checkpoints at steps 50 and 100 is a
  suitable support-verification run when it crosses the peak learning rate and
  completes the configured decay; resume directly from step 50 through step
  100. This verifies bounded training and resume behavior, not full convergence.
- **SFT and PEFT:** Prefer about 100 optimizer steps with warmup and full-horizon
  decay. Use a public dataset name or preset, not its storage location. Save the
  final full-SFT checkpoint when export verification is in scope.
- **SFT export and inference:** Depend on `sft`, export its final full-model
  checkpoint to HF, reload the exported model with Transformers, and run one
  deterministic greedy generation. Store this item as an ordered `commands`
  list containing exactly two strings: the synchronous Slurm export first and
  the maintained `infer.sh --task hf-inference` launcher second. Direct
  `uv run` invocation of the same helper remains valid when no Slurm executor
  is available. Specify an explicit maximum new-token bound, allow natural
  end-of-sequence stopping, and record the actual generated-token count plus
  the literal completion, including whitespace, in `expected_result`.
- **Long-context SFT:** Verify sequence packing and CP together. Record CP only
  when its size is greater than one.
- **Checkpoint resume:** Depend on `pretrain`; load its middle checkpoint
  directly, resume into a distinct new output root, load optimizer and RNG
  state, and compare the first resumed and final steps with the uninterrupted
  reference. For each declared loss sentinel, require this bound:
  `abs(resumed - reference) <= 1e-6 + 0.01 * abs(reference)`. Tighter
  model-specific tolerances are allowed. Do not repeat the pre-checkpoint
  training segment. When the uninterrupted reference intentionally warm-starts
  from `--pretrained_checkpoint`, omit that fallback from the resume command;
  the middle checkpoint already contains the initialized model state. Persist
  each run's post-setup config to its own path so the resume does not overwrite
  the reference evidence.
- **Performance (when present):** Use the exact canonical public performance
  recipe. Keep its bounded mock-data run separate from the real-data functional
  run and state public hardware plus thresholds. A natural-routing MoE
  performance command must warm-start through the card's
  `--pretrained_checkpoint`; a forced-balance benchmark may omit it.

Before adding checkpoint overrides, inspect the selected recipe and its
inherited checkpoint defaults. Keep only values that change the effective
behavior, such as an explicit load/save root, save interval, resume step, or
intentional strictness. For resume, the effect

…(truncated)