Compare Model Checkpoints

Compare Model Checkpoints

Normalize Before Comparing

Identify the exact model and tokenizer revisions, chat template, adapter/merge state, quantization or conversion recipe, prompt set, generation parameters, evaluator, runtime, and hardware. Treat any uncontrolled difference as a confound.

Workflow

1. Preserve every source artifact as immutable, snapshot its provenance with scripts/snapshot-model-provenance.fsx, and write the snapshot outside the artifact directory.
2. Verify that every comparison artifact can be loaded and produces output on the same smoke cases.
3. Use evaluate-language-model for paired quality and behavior evidence.
4. Use benchmark-model-runtime when deployment properties affect the decision.
5. Compare primary metrics, guardrails, per-slice regressions, artifact size, memory, latency, and load reliability.
6. Inspect high-impact case changes rather than choosing by one aggregate score.
7. Copy assets/model-comparison-report.md and give a conditional recommendation when tradeoffs differ by deployment target.

Comparison Rules

• Evaluate the exact artifact that will ship; an unmerged adapter does not prove the merged or quantized export.
• Do not compare stochastic generations without repeated samples or a fixed sampling contract.
• Do not call two checkpoints equivalent because their average scores match; inspect paired disagreements and guardrails.
• Report missing provenance as a finding, not as an implied default.

Resources

• assets/model-comparison-report.md: selection report.
• references/checkpoint-provenance.md: provenance field guide.
• scripts/snapshot-model-provenance.fsx: deterministic local artifact inventory.