kermt-pretrain-scratch
Pretrain a brand-new KERMT model from scratch on a user-provided corpus. Useful
when you want to retrain a model on a custom chemistry domain rather than
extending one of the released checkpoints. Significantly more expensive than
kermt-continue-pretrain — no warm start, so the loss curves need to descend
from scratch over many epochs.
Skill and runtime paths
Set SKILL_DIR to the absolute path of this installed skill directory. Export
KERMT_REPO as the absolute path to the KERMT checkout used for model
execution. The bundled container helper mounts that checkout at
/workspace and this skill at /skill (read-only). Commands inside
the container use /skill/scripts/; defaults are bundled in config/.
Hardware requirements
Same as kermt-continue-pretrain:
GPUs: 1–N CUDA-capable. The runner auto-detects via
torch.cuda.device_count(); --gpus 0,2 overrides. Single-GPU fallback:
--batch_size 32 --save_interval 500. Multi-GPU keeps defaults
(--batch_size 256 etc.). Note: --gpus N uses torch.cuda indexing,
which can differ from nvidia-smi's display order on multi-GPU hosts
(PCI bus vs. CUDA enumeration). To target a specific physical GPU, set
CUDA_VISIBLE_DEVICES before invoking, or run
python -c "import torch; print([torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())])"
to confirm which device you're picking.
VRAM: the default --batch-size 256 is sized for A100-class hardware
(80 GB VRAM). On smaller GPUs, downscale to avoid OOM:
| GPU class |
VRAM |
Suggested --batch-size |
| L4, T4, V100 16 GB |
16–24 GB |
32–64 |
| A100 40 GB, L40, A40 |
40–48 GB |
128 |
| A100 80 GB, H100, H200 |
80 GB |
256 (default) |
These are rough starting points — pass --batch-size N to override.
Disk: tens of GB for shards + vocab + checkpoints, scaled by epochs.
Wall time: this is the big difference. Pretraining from scratch on an
11M-mol corpus at 100 epochs typically takes days even on a multi-GPU box.
The skill prints an estimate before launching; confirm with the user.
When to invoke
- User wants to train a new model on a custom corpus (e.g. domain-specific
chemistry that the released ckpts don't cover).
- User wants to reproduce a pretrain config end-to-end without depending on a
released ckpt.
For continuing an existing released ckpt, use kermt-continue-pretrain. For
adding a cMIM decoder to an encoder-only grover_base ckpt, use
kermt-add-cmim-pretrain.
Inputs
Required:
--csv <path> — the pretrain corpus CSV with a smiles column. Single file
by convention; multi-file corpora deferred. Use --val-csv for a separate
validation set.
--pretrain-target-mode {vocab|cmim|hybrid} — which pretrain objective to
use. No default — must be set explicitly so the user makes an informed
choice:
vocab — original GROVER-style atom + bond vocab prediction (encoder-only
output, lightweight).
cmim — contrastive + SMILES reconstruction objective. Requires building
a SMILES vocab from the corpus.
hybrid — both vocab and contrastive objectives jointly (the
state-of-the-art config from the KERMT manuscript).
Optional:
--val-csv <path> — separate validation CSV. Without it, prepare_data
auto-splits the input by --val-frac 0.1 (random shuffle with --seed).
- Training-hyperparameter overrides:
--epochs N / --batch-size N /
--init-lr F / --max-lr F / --final-lr F / --warmup-epochs F /
--weight-decay F / --dropout F / --save-interval N / --seed N.
Anything not given is filled from config/defaults_pretrain.json.
--vocab-loss-weight F (hybrid only) / --latent-dim N /
--contrastive-temperature F (cmim and hybrid only).
--wandb-project NAME / --wandb-run-name NAME — optional Weights & Biases
logging. When --wandb-project is set, rank 0 logs train/val losses; the run
name is honored only alongside a project. Off by default.
--gpus 0,2 — restrict to a GPU subset.
Workflow
Let $KERMT_REPO be the path to your kermt repo checkout.
Pre-flight: ensure container + system probe (same as
kermt-continue-pretrain step 1). Refuse to proceed if check_system
reports gaps.
Compute run directory.
RUN_DIR=$KERMT_REPO/runs/pretrain-scratch_$(date -u +%Y-%m-%dT%H-%M-%SZ)
Validate the corpus (no ckpt to validate, so this is the only input
check):
"$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> -- \
"python /skill/scripts/check_data.py --mode pretrain --csv /data/<basename>"
Abort on ok: false.
Prepare the data — no vocab pass-through (we want fresh vocab from
corpus):
"$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> --run-dir $RUN_DIR -- \
"python /skill/scripts/prepare_data.py --mode pretrain \\
--csv /data/<basename> --out /runs/data \\
[--val-csv /data/<val-basename>] [--val-frac 0.1] [--seed 0]"
Outputs land at $RUN_DIR/data/prepare_data.json with
vocab_source: "built_fresh".
Estimate runtime + warn loudly. This is critical for pretrain-from-scratch:
- "Pretraining from scratch is days-scale even on multi-GPU; the released
KERMT checkpoints were each trained on millions of molecules for hundreds
of GPU-hours. If you mainly want to leverage existing knowledge for a
downstream task, consider
kermt-continue-pretrain from a released ckpt
instead, which converges in hours instead of days."
- Show the corpus size × epochs × GPU count → estimated wall time.
- Ask for explicit confirmation unless
--yes was given.
Launch the runner detached.
"$SKILL_DIR/scripts/kermt_container.sh" run_detached \\
--name kermt-pretrain-scratch-<ts> \\
--run-dir $RUN_DIR -- \\
"python /skill/scripts/run_pretrain_local.py \\
--from-scratch --pretrain-target-mode <vocab|cmim|hybrid> \\
--prepare-manifest /runs/data/prepare_data.json \\
--out /runs \\
[--epochs N --batch-size N ...]"
Note: NO --ckpt flag (the runner refuses if both --from-scratch and
--ckpt are given). The runner uses the arch group from
config/defaults_pretrain.json to size the model.
Report to the user. Always include all of the following — do not
omit the TensorBoard line under output-length pressure:
- Container name + id
$RUN_DIR/run.json (the manifest with workflow: pretrain-scratch,
from_scratch: true, vocab_check: null, arch from defaults, full
cmd_replay)
- Log file:
$RUN_DIR/logs/pretrain_ddp.log
- TensorBoard:
$RUN_DIR/logs/tb (open with tensorboard --logdir $RUN_DIR/logs/tb)
- Suggest
kermt-monitor <RUN_DIR> for progress.
Hard rules
- Never accept a
--ckpt flag. From-scratch is exclusive with input
ckpt — the runner enforces this; the skill should too.
- Never silently default
--pretrain-target-mode. This is a significant
architectural choice (vocab = lightweight, hybrid = SOTA). Prompt the user
if not given on the CLI.
- Strong warning before launching. From-scratch pretrain is the most
expensive workflow. The user needs to know what they're committing to.
Common errors
--pretrain-target-mode is required when --from-scratch is set → user
forgot the mode flag. Prompt.
--from-scratch is incompatible with --ckpt → user provided both; ask which
one they meant.
defaults_pretrain.json has no arch group → repo state issue (should never
happen on a fresh clone); points the user at running kermt-setup again.
What's in the manifest after a from-scratch run
Same reproducibility fields as continue-pretrain (repo.commit, kermt_image,
cmd_replay, args_applied), plus:
workflow: "pretrain-scratch"
from_scratch: true
inputs.ckpt: null
ckpt_symlink: null
vocab_check: null (not verified — vocab built from corpus is
authoritative for from-scratch)
arch: the values pulled from config/defaults_pretrain.json's
arch group (with any future CLI overrides applied).
Replayability
Same as continue-pretrain: cmd_replay is a copy-pasteable command. If
ok_to_replay: false, the kermt repo working tree was dirty at launch
time — check repo.commit and git checkout it first.
1---2name: kermt-pretrain-scratch3description: Pretrain a fresh KERMT model from scratch on a user-provided corpus. Builds a new vocabulary from the corpus, instantiates the model architecture from defaults, and launches pretrain_ddp.py inside the kermt container (detached for long runs). Unlike kermt-continue-pretrain, no starting checkpoint is loaded — the model is randomly initialized.4license: Apache-2.05compatibility: Requires docker, nvidia-container-toolkit, and a CUDA-capable NVIDIA GPU. Designed for Claude Code, Codex, and Nemotron.6metadata:7 source: https://skillmd.com/skills/nvidia/kermt-pretrain-scratch8 owner: evax@nvidia.com9 classification: workflow-skill10 risk_tier: skill11# Line/token budget: ~210 lines, ~2400 tokens — well within the12# 500-line / 5000-token cap for skill files. Most of the orchestration is13# shared with kermt-continue-pretrain; the differences are documented below.14---1516# kermt-pretrain-scratch1718Pretrain a brand-new KERMT model from scratch on a user-provided corpus. Useful19when you want to retrain a model on a custom chemistry domain rather than20extending one of the released checkpoints. **Significantly more expensive than21`kermt-continue-pretrain`** — no warm start, so the loss curves need to descend22from scratch over many epochs.2324## Skill and runtime paths2526Set `SKILL_DIR` to the absolute path of this installed skill directory. Export27`KERMT_REPO` as the absolute path to the KERMT checkout used for model28execution. The bundled container helper mounts that checkout at29`/workspace` and this skill at `/skill` (read-only). Commands inside30the container use `/skill/scripts/`; defaults are bundled in `config/`.3132## Hardware requirements3334Same as `kermt-continue-pretrain`:3536- **GPUs**: 1–N CUDA-capable. The runner auto-detects via37 `torch.cuda.device_count()`; `--gpus 0,2` overrides. Single-GPU fallback:38 `--batch_size 32 --save_interval 500`. Multi-GPU keeps defaults39 (`--batch_size 256` etc.). Note: `--gpus N` uses **torch.cuda** indexing,40 which can differ from `nvidia-smi`'s display order on multi-GPU hosts41 (PCI bus vs. CUDA enumeration). To target a specific physical GPU, set42 `CUDA_VISIBLE_DEVICES` before invoking, or run43 `python -c "import torch; print([torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())])"`44 to confirm which device you're picking.45- **VRAM**: the default `--batch-size 256` is sized for A100-class hardware46 (80 GB VRAM). On smaller GPUs, downscale to avoid OOM:4748 | GPU class | VRAM | Suggested `--batch-size` |49 |--------------------------|------------|--------------------------|50 | L4, T4, V100 16 GB | 16–24 GB | 32–64 |51 | A100 40 GB, L40, A40 | 40–48 GB | 128 |52 | A100 80 GB, H100, H200 | 80 GB | 256 (default) |5354 These are rough starting points — pass `--batch-size N` to override.55- **Disk**: tens of GB for shards + vocab + checkpoints, scaled by epochs.56- **Wall time**: this is the big difference. Pretraining from scratch on an57 11M-mol corpus at 100 epochs typically takes **days even on a multi-GPU box**.58 The skill prints an estimate before launching; confirm with the user.5960## When to invoke6162- User wants to train a new model on a custom corpus (e.g. domain-specific63 chemistry that the released ckpts don't cover).64- User wants to reproduce a pretrain config end-to-end without depending on a65 released ckpt.6667For continuing an existing released ckpt, use `kermt-continue-pretrain`. For68adding a cMIM decoder to an encoder-only grover_base ckpt, use69`kermt-add-cmim-pretrain`.7071## Inputs7273Required:7475- `--csv <path>` — the pretrain corpus CSV with a `smiles` column. Single file76 by convention; multi-file corpora deferred. Use `--val-csv` for a separate77 validation set.78- `--pretrain-target-mode {vocab|cmim|hybrid}` — which pretrain objective to79 use. **No default** — must be set explicitly so the user makes an informed80 choice:81 - `vocab` — original GROVER-style atom + bond vocab prediction (encoder-only82 output, lightweight).83 - `cmim` — contrastive + SMILES reconstruction objective. Requires building84 a SMILES vocab from the corpus.85 - `hybrid` — both vocab and contrastive objectives jointly (the86 state-of-the-art config from the KERMT manuscript).8788Optional:8990- `--val-csv <path>` — separate validation CSV. Without it, prepare_data91 auto-splits the input by `--val-frac 0.1` (random shuffle with `--seed`).92- Training-hyperparameter overrides: `--epochs N` / `--batch-size N` /93 `--init-lr F` / `--max-lr F` / `--final-lr F` / `--warmup-epochs F` /94 `--weight-decay F` / `--dropout F` / `--save-interval N` / `--seed N`.95 Anything not given is filled from `config/defaults_pretrain.json`.96- `--vocab-loss-weight F` (hybrid only) / `--latent-dim N` /97 `--contrastive-temperature F` (cmim and hybrid only).98- `--wandb-project NAME` / `--wandb-run-name NAME` — optional Weights & Biases99 logging. When `--wandb-project` is set, rank 0 logs train/val losses; the run100 name is honored only alongside a project. Off by default.101- `--gpus 0,2` — restrict to a GPU subset.102103## Workflow104105Let `$KERMT_REPO` be the path to your kermt repo checkout.1061071. **Pre-flight: ensure container + system probe** (same as108 `kermt-continue-pretrain` step 1). Refuse to proceed if `check_system`109 reports gaps.1101112. **Compute run directory.**112 ```113 RUN_DIR=$KERMT_REPO/runs/pretrain-scratch_$(date -u +%Y-%m-%dT%H-%M-%SZ)114 ```1151163. **Validate the corpus** (no ckpt to validate, so this is the only input117 check):118 ```119 "$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> -- \120 "python /skill/scripts/check_data.py --mode pretrain --csv /data/<basename>"121 ```122 Abort on `ok: false`.1231244. **Prepare the data** — no vocab pass-through (we want fresh vocab from125 corpus):126 ```127 "$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> --run-dir $RUN_DIR -- \128 "python /skill/scripts/prepare_data.py --mode pretrain \\129 --csv /data/<basename> --out /runs/data \\130 [--val-csv /data/<val-basename>] [--val-frac 0.1] [--seed 0]"131 ```132 Outputs land at `$RUN_DIR/data/prepare_data.json` with133 `vocab_source: "built_fresh"`.1341355. **Estimate runtime + warn loudly.** This is critical for pretrain-from-scratch:136 - "Pretraining from scratch is days-scale even on multi-GPU; the released137 KERMT checkpoints were each trained on millions of molecules for hundreds138 of GPU-hours. If you mainly want to leverage existing knowledge for a139 downstream task, consider `kermt-continue-pretrain` from a released ckpt140 instead, which converges in hours instead of days."141 - Show the corpus size × epochs × GPU count → estimated wall time.142 - Ask for explicit confirmation unless `--yes` was given.1431446. **Launch the runner detached.**145 ```146 "$SKILL_DIR/scripts/kermt_container.sh" run_detached \\147 --name kermt-pretrain-scratch-<ts> \\148 --run-dir $RUN_DIR -- \\149 "python /skill/scripts/run_pretrain_local.py \\150 --from-scratch --pretrain-target-mode <vocab|cmim|hybrid> \\151 --prepare-manifest /runs/data/prepare_data.json \\152 --out /runs \\153 [--epochs N --batch-size N ...]"154 ```155 Note: NO `--ckpt` flag (the runner refuses if both `--from-scratch` and156 `--ckpt` are given). The runner uses the `arch` group from157 `config/defaults_pretrain.json` to size the model.1581597. **Report to the user.** Always include all of the following — do not160 omit the TensorBoard line under output-length pressure:161 - Container name + id162 - `$RUN_DIR/run.json` (the manifest with `workflow: pretrain-scratch`,163 `from_scratch: true`, `vocab_check: null`, `arch` from defaults, full164 `cmd_replay`)165 - Log file: `$RUN_DIR/logs/pretrain_ddp.log`166 - TensorBoard: `$RUN_DIR/logs/tb` (open with `tensorboard --logdir167 $RUN_DIR/logs/tb`)168 - Suggest `kermt-monitor <RUN_DIR>` for progress.169170## Hard rules171172- **Never accept a `--ckpt` flag.** From-scratch is exclusive with input173 ckpt — the runner enforces this; the skill should too.174- **Never silently default `--pretrain-target-mode`.** This is a significant175 architectural choice (vocab = lightweight, hybrid = SOTA). Prompt the user176 if not given on the CLI.177- **Strong warning before launching.** From-scratch pretrain is the most178 expensive workflow. The user needs to know what they're committing to.179180## Common errors181182- `--pretrain-target-mode is required when --from-scratch is set` → user183 forgot the mode flag. Prompt.184- `--from-scratch is incompatible with --ckpt` → user provided both; ask which185 one they meant.186- `defaults_pretrain.json has no arch group` → repo state issue (should never187 happen on a fresh clone); points the user at running `kermt-setup` again.188189## What's in the manifest after a from-scratch run190191Same reproducibility fields as continue-pretrain (`repo.commit`, `kermt_image`,192`cmd_replay`, `args_applied`), plus:193194- `workflow`: `"pretrain-scratch"`195- `from_scratch`: `true`196- `inputs.ckpt`: `null`197- `ckpt_symlink`: `null`198- `vocab_check`: `null` (not verified — vocab built from corpus is199 authoritative for from-scratch)200- `arch`: the values pulled from `config/defaults_pretrain.json`'s201 `arch` group (with any future CLI overrides applied).202203## Replayability204205Same as continue-pretrain: `cmd_replay` is a copy-pasteable command. If206`ok_to_replay: false`, the kermt repo working tree was dirty at launch207time — check `repo.commit` and `git checkout` it first.