kermt-finetune
Finetune a pretrained KERMT encoder on a user-supplied labeled CSV. The skill
is the workflow orchestrator: validate ckpt, validate data, prepare data,
launch the runner detached, return a run directory + container name.
Skill and runtime paths
Set SKILL_DIR to the absolute path of this installed skill directory. Export
KERMT_REPO as the absolute path to the KERMT checkout used for model
execution. The bundled container helper mounts that checkout at
/workspace and this skill at /skill (read-only). Commands inside
the container use /skill/scripts/; defaults are bundled in config/.
See Released models for checkpoint bundle requirements.
Downloads and local outputs
The optional released-model branch reads config/released_model.json for the
Hugging Face repository, pinned revision, and filenames. The bundled
scripts/fetch_released_model.py downloads the model bundle over HTTPS into
the host directory the user selects. Public models work without credentials;
if HF_TOKEN is set, the container helper forwards it for Hugging Face
authentication. Prepared data, logs, and workflow results go into the chosen
run directory.
Hardware requirements
- GPUs: 1 by default (single-GPU); pass
--gpus 0 (or whichever id) to
select one. For faster training on a multi-GPU host, pass --num-gpus N
(N>1) to run data-parallel DDP across N GPUs — --batch-size is then
per-GPU (effective global batch = batch_size × N).
- VRAM: ≥ 8 GB for the default
batch_size 32 configuration. Lower VRAM
works at smaller batch sizes — pass --batch-size N to override.
- Disk: a few GB per run (checkpoint + features + logs).
- Driver / CUDA: any host supporting CUDA 12.6 (the kermt image base).
kermt-setup validates this up-front.
Inputs
Required:
--csv <path> — labeled CSV. First column is smiles; every other column
is a target.
Checkpoint (optional — defaults to the released model if omitted):
--ckpt <path> — input pretrain checkpoint (grover_base / cmim / hybrid).
The validator refuses already-finetuned ckpts with a redirect to
kermt-infer. If omitted, the skill offers to download the released
pretrained hybrid model nvidia/NV-KERMT-70M-v2 and finetune from it —
see "Resolve & validate the checkpoint" (workflow step 3).
--pretrained-release — explicit opt-in to use the released model without
the interactive prompt (for non-interactive / agent runs). Mutually
exclusive with --ckpt.
--model-dir <dir> — where to save the downloaded bundle (default
$KERMT_REPO/models/NV-KERMT-70M-v2/). An already-complete bundle there is
reused, not re-downloaded.
Optional:
--dataset-type {regression | classification | multiclass} — default
regression (from defaults_finetune.json). Drives loss, metric defaults,
and head initialization. For classification tasks pass
--dataset-type classification.
--targets COL [COL ...] — explicit target column names. If omitted, the
validator auto-detects numeric non-smiles columns and the skill confirms
with the user before proceeding.
--val-csv <path> and --test-csv <path> — user-provided val + test
splits. Either pass both or pass neither (the skill auto-splits using the
configured --split-type).
--split-type {random | scaffold_balanced | index_predetermined} —
default scaffold_balanced from defaults_finetune.json.
random and scaffold_balanced: build the val/test split internally
from the train CSV. No --val-csv / --test-csv needed.
index_predetermined: requires pre-split CSVs passed via
--val-csv + --test-csv (and, separately, per-fold index files —
see kermt/util/utils.split_data). Use this when the dataset ships
its own canonical split (e.g. tests/data/Biogen_for_grover/scaffold/ balance/<endpoint>/{train,val,test}.csv).
--metric NAME — mae (regression default), auc (classification default),
or any name kermt.util.metrics.get_metric_func accepts.
--epochs N / --batch-size N / --init-lr F / --max-lr F /
--final-lr F / --warmup-epochs F / --weight-decay F / --dropout F /
--bond-drop-rate F / --dist-coff F / --early-stop-epoch N /
--seed N — training-hyperparameter overrides. Anything not given is
filled from config/defaults_finetune.json.
--ffn-hidden-size N / --ffn-num-layers N — shared FFN trunk dims.
--ffn-num-task-specific-layers N / --ffn-task-specific-hidden-size H —
per-target FFN heads (default 0 = off; useful for heterogeneous multi-target
finetunes). Both must be set together when N > 0.
--ensemble-size N / --num-folds N — multi-model / k-fold CV. Default 1
each.
--gpus 0 — single GPU id for single-process finetune (default 0). Ignored
when --num-gpus > 1.
--num-gpus N — number of GPUs for data-parallel DDP finetune. Default 1
(single-process, unchanged). N>1 runs main.py finetune with WORLD_SIZE=N
(one process per GPU); --batch-size is per-GPU.
--from-prepare <dir> — skip the prepare step and reuse an existing
prepare_data.json in <dir>. Useful when iterating on hyperparameters.
Workflow
Let $KERMT_REPO be the path to your kermt repo checkout, and assume
kermt-setup has built kermt:latest. All paths below are on the host; the
helper bind-mounts them at known container paths.
Pre-flight: ensure container + system probe.
"$SKILL_DIR/scripts/kermt_container.sh" check_system | python -c "
import json, sys; d = json.load(sys.stdin)
if not d['ok']:
print('System check failed:', d['gaps']); sys.exit(1)
print(f'OK: {len(d[\"gpus\"])} GPU(s); CUDA via container toolkit')
"
Refuse to proceed if ok: false.
Compute run directory.
RUN_DIR=$KERMT_REPO/runs/finetune_$(date -u +%Y-%m-%dT%H-%M-%SZ)
Resolve & validate the checkpoint.
Resolve — only if --ckpt was omitted. Default to the released
pretrained hybrid model nvidia/NV-KERMT-70M-v2:
Validate the resolved (or user-provided) ckpt:
"$SKILL_DIR/scripts/kermt_container.sh" run --ckpt <user-ckpt> -- \
"python /skill/scripts/check_checkpoint.py --mode finetune_init --ckpt /ckpt"
Parse the JSON. Abort on ok: false. The validator rejects already-
finetuned ckpts (has_task_ffn: true) with a redirect to kermt-infer.
Validate the data.
"$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> -- \
"python /skill/scripts/check_data.py --mode finetune --csv /data/<basename> [--targets COL1 COL2 ...]"
If --targets was not given by the user, surface auto_detected_targets
from the JSON and ask the user to confirm before continuing. Abort on
ok: false.
Prepare the data (skip if --from-prepare given).
Pre-flight: check for sibling val.csv / test.csv. Before invoking
prepare_data, inspect the parent directory of <user-csv>. If a
canonical-looking sibling val.csv (or val_*.csv — common variants
include val_T.csv, val_clean.csv) AND a matching test.csv /
test_*.csv exist next to the train CSV, the dataset ships its own
pre-defined split. In that case set --split-type index_predetermined
AND pass --val-csv / --test-csv — otherwise the configured
split_type (default scaffold_balanced) will re-split the train CSV
from scratch and silently discard the user's val/test files. When in
doubt — or when the sibling files use non-canonical suffixes (_T,
_v2, etc.) — surface the situation to the user and ask which they
want.
Quoting target names. If any of the --targets column names
contain shell metacharacters (>, &, |, (, ), $, etc.),
single-quote each one when passing on the CLI to keep the shell from
eating part of the name. Example: --targets 'Log_Caco2_Papp_A>B' 'logD'. The CSV header itself is read directly by the downstream
trainer and is unaffected, but the prepare_data.json manifest's
targets[] field captures whatever the shell delivers — unquoted
metacharacters get truncated there.
Mount note: kermt_container.sh --data <host-csv> mounts the
parent directory of <host-csv> at /data. --val-csv and
--test-csv must therefore reference files in that same parent
directory. If val/test live in a separate directory (e.g. a sibling
splits/ folder), mount the parent of all three using --data <dir>
on a directory rather than a file.
"$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> --run-dir $RUN_DIR -- \
"python /skill/scripts/prepare_data.py --mode finetune \\
--csv /data/<basename> --out /runs/data \\
--split-type <split_type> \\
[--val-csv /data/<val-basename> --test-csv /data/<test-basename>] \\
[--val-frac 0.1 --test-frac 0.1 --seed 0] \\
--targets <COL1> [COL2 ...]"
Outputs land at $RUN_DIR/data/prepare_data.json. For scaffold_balanced
and index_predetermined, prep emits a single clean_full_csv + .npz;
the runner passes them through to main.py finetune which calls
split_data internally with the user-supplied seed.
Estimate runtime + echo applied defaults.
- Finetune wall time is typically minutes-to-hours on 1 GPU.
- Surface a summary of every flag that was filled from the defaults
vs user-supplied, so the user knows what was assumed. The runner
records this in
args_applied.
- Sample message:
"Filling from defaults_finetune.json: epochs=30, batch_size=32, split_type=scaffold_balanced. Override any of these with --<flag>."
Targets confirmation gate (hard requirement). Before launching the
runner, regardless of how the targets list was determined (CLI --targets,
auto-detection in step 4, or a user natural-language request like
"finetune on Caco2 and HLM"), echo the final targets list to the user with
an explicit count:
"Will finetune on N target(s): COL1, COL2, ...". If the user's request
specified a subset that doesn't match this list (e.g., they asked for 2
tasks via natural language but the list still has 4), treat it as a
discrepancy and re-prompt with the diff — never silently proceed on the
wrong target set. Wait for explicit confirmation before launching unless
--yes was given.
Launch the runner detached. (Consistent with the pretrain skills.)
"$SKILL_DIR/scripts/kermt_container.sh" run_detached \\
--name kermt-finetune-<ts> \\
--ckpt <user-ckpt> --run-dir $RUN_DIR -- \\
"python /skill/scripts/run_finetune_local.py \\
--ckpt /ckpt \\
--prepare-manifest /runs/data/prepare_data.json \\
--dataset-type <type> \\
--out /runs \\
[--gpus 0] \\
[--num-gpus N] \\
[--epochs N --batch-size N --init-lr F ...] \\
[--ffn-num-task-specific-layers N --ffn-task-specific-hidden-size H]"
Returns the container name + id + log file path.
Report to the user. Output a short summary:
- Container name + id
$RUN_DIR/run.json (manifest with cmd_replay + image digest)
- Log file:
$RUN_DIR/logs/finetune.log
- TensorBoard:
$RUN_DIR/logs/tb (open with tensorboard --logdir $RUN_DIR/logs/tb)
- Final checkpoints land at
$RUN_DIR/ckpt/fold_0/model_0/model.pt
(best-val) and last_checkpoint.pt (sibling, auto-resume target).
Held-out test predictions + metrics land at
$RUN_DIR/ckpt/fold_0/test_result.csv. Paths vary with --num-folds
/ --ensemble-size.
- To follow progress:
kermt-monitor <RUN_DIR> (one-shot) or
docker logs -f <container-name> (streaming).
- To block until the run finishes (useful for short test runs):
docker wait <container-name> — prints the exit code on completion.
Hard rules
- Never download the released model without consent. When
--ckpt is
omitted, download nvidia/NV-KERMT-70M-v2 only after an explicit user "yes"
or an explicit --pretrained-release flag. --ckpt and
--pretrained-release are mutually exclusive.
- Never modify the user's input ckpt. The runner passes its path via
--checkpoint_path; task/train.py loads it read-only into the model and
attaches a new FFN head. The source file stays untouched.
- Arch comes from the ckpt, not from CLI/defaults. The runner extracts
hidden_size, depth, num_attn_head, activation, embedding_output_type,
self_attention (+ attn_hidden / attn_out when applicable) from the
ckpt's saved_args. There is no --hidden-size flag on this runner.
- Never block on the long-running finetune. The skill launches via
run_detached and returns immediately after step 9. Use kermt-monitor.
- Echo applied defaults back to the user. The
args_applied field of
run.json records every flag's value + source (user / default-config).
Surface a one-line summary of every filled-from-default flag so the user
knows what was assumed.
Common errors
finetune_init requires a pretrain ckpt (grover_base / cmim / hybrid) →
the ckpt you passed is already finetuned (has task FFN heads). Pick a
pretrain ckpt instead, or use kermt-infer if you want to run
predictions with the existing finetuned model. To resume a finetune on
the SAME dataset, bypass the skill and call
python main.py finetune --checkpoint_path <ckpt> ... directly — the
agent skill doesn't support resume because saved-task identity
can't be machine-verified against the new training data.
prepare_data manifest reports ok=False → check errors for the failed
step (typically clean_smiles or save_features). Fix and re-run.
ffn_num_task_specific_layers=N>0 but ffn_task_specific_hidden_size is unset
→ MTL heads need an explicit hidden size. Pass --ffn-task-specific-hidden-size H.
finetune is single-GPU (from --gpus 0,1) → --gpus selects one device
for single-process finetune. For multi-GPU, use --num-gpus N (DDP) instead.
Replayability
The run.json cmd_replay field is a single-line command that re-runs the
finetune with the same inputs, hyperparameters, and arch. To replay inside
the kermt container:
$(jq -r .cmd_replay $RUN_DIR/run.json)
If ok_to_replay: false in the manifest (because the kermt repo working
tree was dirty at launch time), the replay may not be bit-exact — pin the
exact commit via the repo.commit field and git checkout it
first.
1---2name: kermt-finetune3description: Finetune a pretrained KERMT encoder on a labeled CSV. Validate the checkpoint and data, prepare features, and run containerized training. Use a local checkpoint or optionally download a pinned Hugging Face model bundle using HF_TOKEN if configured. Write model bundles, prepared data, logs, and trained models to user-selected host directories.4license: Apache-2.05compatibility: Requires docker, nvidia-container-toolkit, and a CUDA-capable NVIDIA GPU. Designed for Claude Code, Codex, and Nemotron.6metadata:7 source: https://skillmd.com/skills/nvidia/kermt-finetune8 owner: evax@nvidia.com9 classification: workflow-skill10 risk_tier: skill11# Line/token budget: targets ~250 lines / ~3000 tokens — within the12# 500-line / 5000-token cap for skill files.13---1415# kermt-finetune1617Finetune a pretrained KERMT encoder on a user-supplied labeled CSV. The skill18is the workflow orchestrator: validate ckpt, validate data, prepare data,19launch the runner detached, return a run directory + container name.2021## Skill and runtime paths2223Set `SKILL_DIR` to the absolute path of this installed skill directory. Export24`KERMT_REPO` as the absolute path to the KERMT checkout used for model25execution. The bundled container helper mounts that checkout at26`/workspace` and this skill at `/skill` (read-only). Commands inside27the container use `/skill/scripts/`; defaults are bundled in `config/`.28See [Released models](references/released-models.md) for checkpoint bundle requirements.2930## Downloads and local outputs3132The optional released-model branch reads `config/released_model.json` for the33Hugging Face repository, pinned revision, and filenames. The bundled34`scripts/fetch_released_model.py` downloads the model bundle over HTTPS into35the host directory the user selects. Public models work without credentials;36if `HF_TOKEN` is set, the container helper forwards it for Hugging Face37authentication. Prepared data, logs, and workflow results go into the chosen38run directory.3940## Hardware requirements4142- **GPUs**: 1 by default (single-GPU); pass `--gpus 0` (or whichever id) to43 select one. For faster training on a multi-GPU host, pass `--num-gpus N`44 (N>1) to run data-parallel DDP across N GPUs — `--batch-size` is then45 per-GPU (effective global batch = batch_size × N).46- **VRAM**: ≥ 8 GB for the default `batch_size 32` configuration. Lower VRAM47 works at smaller batch sizes — pass `--batch-size N` to override.48- **Disk**: a few GB per run (checkpoint + features + logs).49- **Driver / CUDA**: any host supporting CUDA 12.6 (the kermt image base).50 `kermt-setup` validates this up-front.5152## Inputs5354Required:5556- `--csv <path>` — labeled CSV. First column is `smiles`; every other column57 is a target.5859Checkpoint (optional — defaults to the released model if omitted):6061- `--ckpt <path>` — input pretrain checkpoint (grover_base / cmim / hybrid).62 The validator refuses already-finetuned ckpts with a redirect to63 `kermt-infer`. **If omitted**, the skill offers to download the released64 pretrained hybrid model **nvidia/NV-KERMT-70M-v2** and finetune from it —65 see "Resolve & validate the checkpoint" (workflow step 3).66- `--pretrained-release` — explicit opt-in to use the released model without67 the interactive prompt (for non-interactive / agent runs). Mutually68 exclusive with `--ckpt`.69- `--model-dir <dir>` — where to save the downloaded bundle (default70 `$KERMT_REPO/models/NV-KERMT-70M-v2/`). An already-complete bundle there is71 reused, not re-downloaded.7273Optional:7475- `--dataset-type {regression | classification | multiclass}` — default76 `regression` (from `defaults_finetune.json`). Drives loss, metric defaults,77 and head initialization. For classification tasks pass78 `--dataset-type classification`.7980- `--targets COL [COL ...]` — explicit target column names. If omitted, the81 validator auto-detects numeric non-smiles columns and the skill confirms82 with the user before proceeding.83- `--val-csv <path>` and `--test-csv <path>` — user-provided val + test84 splits. Either pass both or pass neither (the skill auto-splits using the85 configured `--split-type`).86- `--split-type {random | scaffold_balanced | index_predetermined}` —87 default `scaffold_balanced` from `defaults_finetune.json`.88 - `random` and `scaffold_balanced`: build the val/test split internally89 from the train CSV. No `--val-csv` / `--test-csv` needed.90 - `index_predetermined`: **requires** pre-split CSVs passed via91 `--val-csv` + `--test-csv` (and, separately, per-fold index files —92 see `kermt/util/utils.split_data`). Use this when the dataset ships93 its own canonical split (e.g. `tests/data/Biogen_for_grover/scaffold/94 balance/<endpoint>/{train,val,test}.csv`).95- `--metric NAME` — `mae` (regression default), `auc` (classification default),96 or any name `kermt.util.metrics.get_metric_func` accepts.97- `--epochs N` / `--batch-size N` / `--init-lr F` / `--max-lr F` /98 `--final-lr F` / `--warmup-epochs F` / `--weight-decay F` / `--dropout F` /99 `--bond-drop-rate F` / `--dist-coff F` / `--early-stop-epoch N` /100 `--seed N` — training-hyperparameter overrides. Anything not given is101 filled from `config/defaults_finetune.json`.102- `--ffn-hidden-size N` / `--ffn-num-layers N` — shared FFN trunk dims.103- `--ffn-num-task-specific-layers N` / `--ffn-task-specific-hidden-size H` —104 per-target FFN heads (default 0 = off; useful for heterogeneous multi-target105 finetunes). Both must be set together when N > 0.106- `--ensemble-size N` / `--num-folds N` — multi-model / k-fold CV. Default 1107 each.108- `--gpus 0` — single GPU id for single-process finetune (default 0). Ignored109 when `--num-gpus > 1`.110- `--num-gpus N` — number of GPUs for data-parallel DDP finetune. Default 1111 (single-process, unchanged). N>1 runs `main.py finetune` with `WORLD_SIZE=N`112 (one process per GPU); `--batch-size` is per-GPU.113- `--from-prepare <dir>` — skip the prepare step and reuse an existing114 `prepare_data.json` in `<dir>`. Useful when iterating on hyperparameters.115116## Workflow117118Let `$KERMT_REPO` be the path to your kermt repo checkout, and assume119`kermt-setup` has built `kermt:latest`. All paths below are on the host; the120helper bind-mounts them at known container paths.1211221. **Pre-flight: ensure container + system probe.**123 ```124 "$SKILL_DIR/scripts/kermt_container.sh" check_system | python -c "125 import json, sys; d = json.load(sys.stdin)126 if not d['ok']:127 print('System check failed:', d['gaps']); sys.exit(1)128 print(f'OK: {len(d[\"gpus\"])} GPU(s); CUDA via container toolkit')129 "130 ```131 Refuse to proceed if `ok: false`.1321332. **Compute run directory.**134 ```135 RUN_DIR=$KERMT_REPO/runs/finetune_$(date -u +%Y-%m-%dT%H-%M-%SZ)136 ```1371383. **Resolve & validate the checkpoint.**139140 **Resolve — only if `--ckpt` was omitted.** Default to the released141 pretrained hybrid model **nvidia/NV-KERMT-70M-v2**:142 - **Consent gate.** Unless `--pretrained-release` was passed, ask the user:143 "No checkpoint given — download the released model nvidia/NV-KERMT-70M-v2144 (NVIDIA Open Model License, https://huggingface.co/nvidia/NV-KERMT-70M-v2)145 and finetune from it? [y/N]". **Never download without an explicit yes**146 (or `--pretrained-release`). If both `--ckpt` and `--pretrained-release`147 are given, abort — they conflict.148 - **Save location.** Default `$KERMT_REPO/models/NV-KERMT-70M-v2/`; honor149 `--model-dir <dir>` if given. An already-complete bundle is reused.150 - **Download** (foreground; ~282 MB on first fetch):151 ```152 "$SKILL_DIR/scripts/kermt_container.sh" run --model-dir <save-dir> -- \153 "python /skill/scripts/fetch_released_model.py --out /model"154 ```155 Parse the JSON; abort on `ok: false` (surface `errors`). On success set156 `<user-ckpt> = <save-dir>/kermt_contrastive_v2.0.pt`.157158 **Validate** the resolved (or user-provided) ckpt:159 ```160 "$SKILL_DIR/scripts/kermt_container.sh" run --ckpt <user-ckpt> -- \161 "python /skill/scripts/check_checkpoint.py --mode finetune_init --ckpt /ckpt"162 ```163 Parse the JSON. Abort on `ok: false`. The validator rejects already-164 finetuned ckpts (`has_task_ffn: true`) with a redirect to `kermt-infer`.1651664. **Validate the data.**167 ```168 "$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> -- \169 "python /skill/scripts/check_data.py --mode finetune --csv /data/<basename> [--targets COL1 COL2 ...]"170 ```171 If `--targets` was not given by the user, surface `auto_detected_targets`172 from the JSON and ask the user to confirm before continuing. Abort on173 `ok: false`.1741755. **Prepare the data** (skip if `--from-prepare` given).176177 **Pre-flight: check for sibling val.csv / test.csv.** Before invoking178 prepare_data, inspect the parent directory of `<user-csv>`. If a179 canonical-looking sibling `val.csv` (or `val_*.csv` — common variants180 include `val_T.csv`, `val_clean.csv`) AND a matching `test.csv` /181 `test_*.csv` exist next to the train CSV, the dataset ships its own182 pre-defined split. **In that case set `--split-type index_predetermined`183 AND pass `--val-csv` / `--test-csv`** — otherwise the configured184 `split_type` (default `scaffold_balanced`) will re-split the train CSV185 from scratch and silently discard the user's val/test files. When in186 doubt — or when the sibling files use non-canonical suffixes (`_T`,187 `_v2`, etc.) — surface the situation to the user and ask which they188 want.189190 **Quoting target names.** If any of the `--targets` column names191 contain shell metacharacters (`>`, `&`, `|`, `(`, `)`, `$`, etc.),192 single-quote each one when passing on the CLI to keep the shell from193 eating part of the name. Example: `--targets 'Log_Caco2_Papp_A>B'194 'logD'`. The CSV header itself is read directly by the downstream195 trainer and is unaffected, but the prepare_data.json manifest's196 `targets[]` field captures whatever the shell delivers — unquoted197 metacharacters get truncated there.198199 **Mount note:** `kermt_container.sh --data <host-csv>` mounts the200 parent directory of `<host-csv>` at `/data`. `--val-csv` and201 `--test-csv` must therefore reference files in that same parent202 directory. If val/test live in a separate directory (e.g. a sibling203 `splits/` folder), mount the parent of all three using `--data <dir>`204 on a directory rather than a file.205206 ```207 "$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> --run-dir $RUN_DIR -- \208 "python /skill/scripts/prepare_data.py --mode finetune \\209 --csv /data/<basename> --out /runs/data \\210 --split-type <split_type> \\211 [--val-csv /data/<val-basename> --test-csv /data/<test-basename>] \\212 [--val-frac 0.1 --test-frac 0.1 --seed 0] \\213 --targets <COL1> [COL2 ...]"214 ```215 Outputs land at `$RUN_DIR/data/prepare_data.json`. For `scaffold_balanced`216 and `index_predetermined`, prep emits a single `clean_full_csv` + `.npz`;217 the runner passes them through to `main.py finetune` which calls218 `split_data` internally with the user-supplied seed.2192206. **Estimate runtime + echo applied defaults.**221 - Finetune wall time is typically minutes-to-hours on 1 GPU.222 - Surface a summary of every flag that was filled from the defaults223 vs user-supplied, so the user knows what was assumed. The runner224 records this in `args_applied`.225 - Sample message:226 `"Filling from defaults_finetune.json: epochs=30, batch_size=32,227 split_type=scaffold_balanced. Override any of these with --<flag>."`2282297. **Targets confirmation gate (hard requirement).** Before launching the230 runner, regardless of how the targets list was determined (CLI `--targets`,231 auto-detection in step 4, or a user natural-language request like232 "finetune on Caco2 and HLM"), echo the final targets list to the user with233 an explicit count:234 `"Will finetune on N target(s): COL1, COL2, ..."`. If the user's request235 specified a subset that doesn't match this list (e.g., they asked for 2236 tasks via natural language but the list still has 4), treat it as a237 discrepancy and re-prompt with the diff — never silently proceed on the238 wrong target set. Wait for explicit confirmation before launching unless239 `--yes` was given.2402418. **Launch the runner detached.** (Consistent with the pretrain skills.)242 ```243 "$SKILL_DIR/scripts/kermt_container.sh" run_detached \\244 --name kermt-finetune-<ts> \\245 --ckpt <user-ckpt> --run-dir $RUN_DIR -- \\246 "python /skill/scripts/run_finetune_local.py \\247 --ckpt /ckpt \\248 --prepare-manifest /runs/data/prepare_data.json \\249 --dataset-type <type> \\250 --out /runs \\251 [--gpus 0] \\252 [--num-gpus N] \\253 [--epochs N --batch-size N --init-lr F ...] \\254 [--ffn-num-task-specific-layers N --ffn-task-specific-hidden-size H]"255 ```256 Returns the container name + id + log file path.2572589. **Report to the user.** Output a short summary:259 - Container name + id260 - `$RUN_DIR/run.json` (manifest with cmd_replay + image digest)261 - Log file: `$RUN_DIR/logs/finetune.log`262 - TensorBoard: `$RUN_DIR/logs/tb` (open with `tensorboard --logdir263 $RUN_DIR/logs/tb`)264 - Final checkpoints land at `$RUN_DIR/ckpt/fold_0/model_0/model.pt`265 (best-val) and `last_checkpoint.pt` (sibling, auto-resume target).266 Held-out test predictions + metrics land at267 `$RUN_DIR/ckpt/fold_0/test_result.csv`. Paths vary with `--num-folds`268 / `--ensemble-size`.269 - To follow progress: `kermt-monitor <RUN_DIR>` (one-shot) or270 `docker logs -f <container-name>` (streaming).271 - To block until the run finishes (useful for short test runs):272 `docker wait <container-name>` — prints the exit code on completion.273274## Hard rules275276- **Never download the released model without consent.** When `--ckpt` is277 omitted, download `nvidia/NV-KERMT-70M-v2` only after an explicit user "yes"278 or an explicit `--pretrained-release` flag. `--ckpt` and279 `--pretrained-release` are mutually exclusive.280- **Never modify the user's input ckpt.** The runner passes its path via281 `--checkpoint_path`; `task/train.py` loads it read-only into the model and282 attaches a new FFN head. The source file stays untouched.283- **Arch comes from the ckpt, not from CLI/defaults.** The runner extracts284 `hidden_size`, `depth`, `num_attn_head`, `activation`, `embedding_output_type`,285 `self_attention` (+ `attn_hidden` / `attn_out` when applicable) from the286 ckpt's saved_args. There is no `--hidden-size` flag on this runner.287- **Never block on the long-running finetune.** The skill launches via288 `run_detached` and returns immediately after step 9. Use `kermt-monitor`.289- **Echo applied defaults back to the user.** The `args_applied` field of290 `run.json` records every flag's value + source (user / default-config).291 Surface a one-line summary of every filled-from-default flag so the user292 knows what was assumed.293294## Common errors295296- `finetune_init requires a pretrain ckpt (grover_base / cmim / hybrid)` →297 the ckpt you passed is already finetuned (has task FFN heads). Pick a298 pretrain ckpt instead, or use `kermt-infer` if you want to run299 predictions with the existing finetuned model. To resume a finetune on300 the SAME dataset, bypass the skill and call301 `python main.py finetune --checkpoint_path <ckpt> ...` directly — the302 agent skill doesn't support resume because saved-task identity303 can't be machine-verified against the new training data.304- `prepare_data manifest reports ok=False` → check `errors` for the failed305 step (typically clean_smiles or save_features). Fix and re-run.306- `ffn_num_task_specific_layers=N>0 but ffn_task_specific_hidden_size is unset`307 → MTL heads need an explicit hidden size. Pass `--ffn-task-specific-hidden-size H`.308- `finetune is single-GPU` (from `--gpus 0,1`) → `--gpus` selects one device309 for single-process finetune. For multi-GPU, use `--num-gpus N` (DDP) instead.310311## Replayability312313The `run.json` `cmd_replay` field is a single-line command that re-runs the314finetune with the same inputs, hyperparameters, and arch. To replay inside315the kermt container:316317```bash318$(jq -r .cmd_replay $RUN_DIR/run.json)319```320321If `ok_to_replay: false` in the manifest (because the kermt repo working322tree was dirty at launch time), the replay may not be bit-exact — pin the323exact commit via the `repo.commit` field and `git checkout` it324first.