Check fine-tuning readiness
This skill checks preparation only. It provides no training recipe. A
ready result always has training_validated: false. Multi-node training is
unsupported here; do not invent a distributed command or report a successful
fine-tune from this check.
Preconditions
- Read the readiness guide. Run from the repository root as the intended model consumer on the selected compute host, not the login node. The validator checks local host identity against the scheduler record; do not bypass a mismatch.
- Obtain authorization for the target and read-only scheduler queries. Specify one node and one partition. An IDLE node with an unallocated GPU is required; this conservative check does not accept mixed, draining or flagged nodes.
- Follow manage-model-cache for an approved
shared cache, immutable revision and complete reviewed required-file list.
Supported inputs are Hugging Face
tokenizer.jsonplus safetensors weights; list every extra model-specific asset. Indexed weights require every shard. - Select a site-reviewed fine-tuning image pinned by digest, using Pyxis syntax
registry#repository@sha256:<digest>. Do not use mutable tags, credentials in the reference, or substitute an unreviewed image to get a pass.
Readiness check
Set the variables to reviewed values. For sharded weights, replace the final
file argument with model.safetensors.index.json; the validator follows its
weight map. Add every additional file needed by the selected model.
python3 scripts/validation/validate_finetune.py --json \
--node "$NODE" --partition "$PARTITION" \
--container-image "$TRAINING_IMAGE" \
--cache-dir "$HF_HUB_CACHE" --repo-id "$MODEL_REPO" --revision "$MODEL_REVISION" \
--require-file config.json --require-file tokenizer.json \
--require-file tokenizer_config.json --require-file model.safetensors
| Result | Next action |
|---|---|
Exit 0, status: ready, ok: true |
Report readiness only, including that container execution and training remain unproven. |
Exit 1, status: not_ready |
Report every errors[] stage. Repair the named prerequisite through its approved procedure; do not drain nodes, install software or download models implicitly. |
Exit 2, status: bad_input |
Correct explicit inputs or unsupported flags using --help. No scheduler commands run for bad input. |
Optional one-GPU probe
Off by default. Only with explicit authorization to allocate a GPU and run the
reviewed image, repeat the command with --gpu-smoke. This can download the
container image, but its scheduler window is only five seconds, so pre-pull the
pinned image on the node first. It submits one bounded, single-node, single-GPU
nvidia-smi job after all readiness gates pass. It does not load the model or train.
gpu_smoke_ok: true proves only the container GPU probe, not cache access
inside the container, framework compatibility, learning or checkpoint recovery.
Keep credentials private, mount shared inputs read-only in any future reviewed recipe, and keep outputs/checkpoints outside the input cache. A readable cache, a pinned image string, and a GPU probe cannot certify a training outcome.