Select And Verify Best Checkpoint

Select and Verify Best Checkpoint

Treat checkpoint identity as an evidence requirement, not a configuration assumption.

Configure selection explicitly

Set evaluation and saving to compatible cadences. Specify the metric, whether greater is better, and best-model loading. Ensure the configured metric name matches the emitted evaluation key; account for frameworks that add an eval_ prefix.

Verify trainer state

After training, run:

python scripts/verify_checkpoint.py output/trainer_state.json \
  --metric eval_loss \
  --expected-eval-steps 50,100,150 \
  --evaluation-manifest held-out-run.json \
  --loaded-checkpoint-field loadedCheckpoint \
  --output checkpoint-proof.json

The script fails unless:

• the expected evaluation steps are present;
• best_metric and best_model_checkpoint are non-null;
• the recorded metric equals the optimum in evaluation history;
• the best checkpoint step matches the optimum step;
• the held-out manifest identifies that exact checkpoint.

Use --greater-is-better for maximized metrics and repeat --required-file to require adapter or model files inside the best checkpoint.

Preserve proof

Record the trainer state hash, checkpoint path and hash inventory, best metric and step, evaluation manifest, and evaluation command. If ties exist, report the framework's selected tied checkpoint rather than pretending the optimum is unique. Never claim held-out results came from the best checkpoint without an explicit load identity.