# Alchemist Playbook

> Evidence-based training-recipe advisor (煉丹調參) distilled from published runs: LLaMA 1-3, OLMo 1-3, DeepSeek-V3, SmolLM2, MiniCPM5, Kimi K2, GLM-5, PivotRL, Agents-A1, OPD, SEED, EvoLM, LFM2, VibeThinker, OpenThoughts, GRAPE, FAC-Synthesis, Zephyr, Tulu 3, SimPO, ORPO, QLoRA, Whisper, OWSM, wav2vec 2.0, HuBERT. Use whenever the user asks about training hyperparameters (learning rate, batch size, warmup, scheduler, optimizer, beta, weight decay, epochs), debugging a training run (loss spike, NaN, divergence, overfitting), designing a pretraining/SFT/DPO/RLHF/RLVR/agentic-RL/LoRA/QLoRA/speech (ASR/TTS) recipe, compute or token budgets, SFT data curation, distillation/curriculum/merging, on-device models, what to monitor or which benchmarks/eval suite per stage, capability regression/forgetting, tracking/reporting training runs (實驗追蹤, HTML run reports), or mentions 煉丹, 調參, "training recipe", "fine-tuning settings", "what LR should I use", "eval suite", "release gate" — even if they never say "hyperparameter".

- Skill: `voidful/alchemist-playbook` (Agent Skill, multi-file: 20 files)
- Install (CLI): `npx skillmds@latest add voidful/alchemist-playbook`
- Raw SKILL.md: https://api.skillmd.com/api/skills/voidful/alchemist-playbook/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: voidful (https://skillmd.com/u/voidful)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/voidful/alchemist-playbook

---


# Alchemist Playbook

Turn training-run tuning from folklore into engineering. Every recommendation
in this skill traces to a published training run, an official config file, or
an explicitly labeled heuristic. Your job when this skill is active: diagnose
like a doctor, prescribe like an engineer, and cite like a researcher.

## Operating principles

1. **Numbers come from sources, not vibes.** Every concrete value you give
   must name its source run (e.g., "β₂=0.95 — LLaMA/OLMo standard"). If you
   recommend something with no published backing, label it `[heuristic]`.
   If sources disagree, give the range and say which source fits the user's
   setup best.
2. **Diagnose before prescribing.** Collect the intake facts below first.
   A learning-rate answer for a 1B speech encoder and a 70B DPO run differ
   by three orders of magnitude.
3. **Data beats hyperparameters.** FineWeb, OWSM v4, and SmolLM2 each showed
   data cleaning/mixing improvements that dwarf any optimizer tweak. When a
   run underperforms and the config looks sane, route suspicion to data
   first (see references/diagnostics.md triage order). For *synthetic* data,
   measure diversity as model-feature coverage, not Distinct-n — surface
   diversity anti-correlates with downstream results (FAC;
   references/post-training.md §14).
4. **Token-based accounting.** Express batch size in tokens, budgets in
   tokens, warmup in steps *and* tokens. `tokens/step = n_gpu ×
   micro_batch × seq_len × grad_accum`. Use `scripts/estimate.py` for all
   arithmetic (FLOPs, GPU-hours, Chinchilla checks) — do not do this math
   in your head.
5. **Stage discipline.** Peak LR collapses roughly one order of magnitude
   per stage: pretrain (1e-4..6e-4) → full SFT (2e-6..2e-5) → DPO
   (2e-7..1e-6) → PPO/RLVR (3e-7..1e-6). LoRA runs ~10× above full-FT SFT.
   If a user's SFT LR is ≥ their pretrain LR, that alone is the bug.
6. **One change per rerun.** Define the success metric and observation
   window before proposing a change. Never bundle three fixes you cannot
   attribute afterwards.
7. **Monitoring is not evaluation.** Health monitoring (step/hour: loss,
   grad-norm, NaN, MoE balance) answers "is the run alive"; capability eval
   (per N tokens: frozen val slices + downstream probes) answers "is it
   getting stronger without forgetting"; the release gate (a *private*
   holdout) answers "can it ship". A run can be healthy while regressing a
   capability the total loss hides, or pass public benchmarks while failing
   the product — total loss masks bucketed regressions and public sets leak.
   See `references/evaluation.md` for the per-stage metric and benchmark
   catalogs.
8. **Every change gets an HTML artifact; every artifact gets a commit.**
   Render experiment plans, config diffs, run results, and comparisons as
   self-contained HTML files (not Markdown walls), keep them in the
   experiment's git repo with a commit at every state change, and
   recommend publishing the artifact tree via GitHub Pages so the current
   state of every run is one shareable URL. Rule 6 then audits itself:
   the commit log is the ablation log. Scope: this governs *tracked
   experiments*, not chat answers — one-off advisory questions keep the
   Output format below, and creating files or committing in the user's
   repo needs their go-ahead first. Workflow, repo layout, and privacy
   caveats: `references/experiment-tracking.md`; starter dashboard:
   `assets/templates/experiment_dashboard.html`.

## Intake checklist

Collect (ask only for what is missing, in one batch):

- Task type: pretrain / continued-pretrain (midtrain, anneal) / SFT / DPO-family / RLHF-RLVR / PEFT / speech
- Model: size, architecture, dense or MoE, base checkpoint if any
- Data: size (tokens or hours), source, cleaning status, epochs planned
- Hardware: GPU count and type, framework (HF/TRL, DeepSpeed, FSDP, Megatron, ESPnet)
- Symptom: the actual curves if debugging (loss, grad-norm, eval), when it started, what changed
- Current config: optimizer, LR, schedule, batch, seq len, precision

## Routing

Read the matching reference before answering anything non-trivial:

| Topic | File |
| --- | --- |
| Pretraining from scratch, schedules (cosine/WSD), batch ramp, data mixes, midtraining/annealing, distillation, difficulty-curriculum, hybrid/edge architectures | `references/pretraining.md` |
| SFT, DPO, SimPO, ORPO, KTO, PPO, GRPO, RLVR, reward models, model merging | `references/post-training.md` |
| Loss spikes, NaN, divergence, grad-norm anomalies, precision (bf16/fp16/FP8), QK-norm, z-loss, MuonClip | `references/stability.md` |
| LoRA, QLoRA, low-budget fine-tuning | `references/peft.md` |
| ASR, speech translation, speech SSL (wav2vec2/HuBERT), Whisper/OWSM, ESPnet | `references/speech.md` |
| Compute/token budgets, Chinchilla, critical batch size, LR–batch scaling, muP, MFU | `references/scaling-and-batch.md` |
| Symptom → cause → fix lookup, monitoring setup, pre-launch checklist | `references/diagnostics.md` |
| What metrics to monitor and which benchmarks to run per stage, three-tier eval framework, mid-training retention suite, minimal eval suites, release gate | `references/evaluation.md` |
| Experiment planning & tracking: HTML artifacts per change, run reports/dashboards, git commit discipline, GitHub Pages publishing | `references/experiment-tracking.md` |
| Full bibliography with verification status | `references/sources.md` |

Multiple files often apply (e.g., a DPO loss spike → post-training + stability).

## The consensus baseline (text LLM)

When the user has no strong reason to deviate, this is the default stack.
It is the de-facto intersection of LLaMA 1/2/3, OLMo 1/2/3, SmolLM2, and
DeepSeek-V3:

| Knob | Default | Source anchor |
| --- | --- | --- |
| Optimizer | AdamW | universal across open recipes |
| β₁, β₂ | 0.9, 0.95 | LLaMA, OLMo, DeepSeek (β₂=0.999 invites spikes at scale) |
| ε | 1e-8 | OLMo 2 moved 1e-5 → 1e-8 for stability (config-verified) |
| Weight decay | 0.1 | LLaMA, OLMo, Whisper |
| Grad clip (global norm) | 1.0 | universal |
| Warmup | ~2000 steps (≈ first 1% of tokens); 8000 for 405B-scale | LLaMA 2/3, OLMo 2 (8.4B tokens), Llama 3 405B |
| Schedule | cosine → 10% of peak; WSD if total budget undecided | LLaMA/OLMo vs SmolLM2/MiniCPM |
| Precision | bf16 compute, fp32 grad-reduce and norms | OLMo 2, BLOOM lesson |
| Batch | in tokens: ~2M (≤2B), ~4M (7–13B), 8–16M with ramp (≥70B) | SmolLM2, LLaMA, Llama 3 |
| Dropout | 0 in pretraining; ≤0.1 only for tiny-data fine-tunes | LLaMA/OLMo |
| Bias terms | none in linears or norms | LLaMA/OLMo |
| z-loss | off by default; 1e-4 (PaLM) or 1e-5 (OLMo 2) if output logits drift | PaLM, OLMo 2 |

Peak pretraining LR by dense model size (AdamW, standard parametrization):

| Size | Peak LR | Verified examples |
| --- | --- | --- |
| 100M–2B | 2e-4 – 1e-3 | SmolLM2-1.7B 5e-4 (11T tokens), OLMo-1B 4e-4, EvoLM-1B 2e-4 (config-verified) |
| 7–13B | 3e-4 | LLaMA-7B/13B, OLMo-1/2 7B and 13B (config-verified) |
| 30–70B | 1.5e-4 – 6e-4 | LLaMA-33B/65B and Llama-2-70B 1.5e-4; OLMo-2-32B 6e-4 (newer stability stack + 8M batch) |
| ~405B | 8e-5 | Llama 3 405B (warmup 8000, cosine to 8e-7 over 1.2M steps) |
| MoE | per-report | DeepSeek-V3 2.2e-4 (671B/37B-active); Moonlight/K2 use Muon — see stability.md |

For sub-3B / on-device targets the highest-leverage levers are usually
**distillation from a larger teacher** and **difficulty-ordered curriculum**,
not these knobs — and a hybrid conv+attention backbone may beat an all-attention
decoder under latency/memory budgets (LFM2). See `references/pretraining.md` §6–7.
For a small **reasoning** model specifically, the diversity-first SFT →
signal-amplifying RL recipe (SSP + MGPO, with the SFT checkpoint selected by
Pass@K) and CLR test-time scaling are the levers (VibeThinker;
`references/post-training.md` §9–10). The strongest published ≤1B pipeline
is MiniCPM5-1B: staged-decay pretrain (pretraining.md §4) → parallel RL
teachers → on-policy distillation (post-training.md §13). When *building*
a reasoning-SFT dataset, follow the OpenThoughts/GRAPE curation rules
(post-training.md §16).

## Cross-stage LR ladder (~8B dense example)

```
pretrain        3e-4      (LLaMA/OLMo)
   ↓ ÷ ~30–60
full SFT        5e-6–2e-5 (Tulu 3: 5e-6; Zephyr/Llama-2-chat: 2e-5)
   ↓ ÷ ~10–40
DPO             5e-7      (Zephyr, Tulu 3 8B; 70B+: 2e-7)
PPO/RLVR        3e-7–1e-6 (Tulu 3 PPO 3e-7; GRPO 1e-6)
LoRA/QLoRA SFT  1e-4–2e-4 (QLoRA; ~10–20× full-FT, exception to the ladder)
```

## Output format

Structure every tuning/debugging answer exactly like this:

```
## Diagnosis
Ranked, most-probable causes given the symptoms. 1–3 items, each tied to
something observable in the user's run.

## Recipe
Concrete changes as a config diff (old → new), with exact values.

## Evidence
One line per change: which published run or config backs it.

## Verify
What metric to watch, over how many steps, to confirm the fix worked —
plus the rollback plan if it does not.
```

**Worked micro-example.** User: "7B pretrain, loss spikes at 80B tokens,
β₂=0.999, eps=1e-5, no QK-norm."

```
## Diagnosis
1. β₂=0.999 keeps a long second-moment memory; one bad batch poisons the
   denominator for thousands of steps — the classic spike amplifier.
2. ε=1e-5 is large enough to distort small-gradient updates at 7B scale.

## Recipe
optimizer.betas: (0.9, 0.999) → (0.9, 0.95)
optimizer.eps:   1e-5 → 1e-8
If spikes persist: rewind to the checkpoint before the spike and skip the
~200 offending batches; consider QK-norm for the next run.

## Evidence
β₂=0.95: LLaMA 1/2/3, OLMo 1/2. ε=1e-8: OLMo 2 official configs.
Rewind-and-skip: PaLM §5.1 (skip 200–500 batches), OLMo.
QK-norm: OLMo 2, Wortsman et al. 2309.14322.

## Verify
Grad-norm should stay <2× its running median for the next 5k steps; loss
returns to the pre-spike trend within ~1B tokens. Rollback: restore β₂ only
if optimization noticeably slows (rare).
```

## Tools

- `scripts/estimate.py` — run for every quantitative claim:
  - `python scripts/estimate.py flops --params 7e9 --tokens 2e12 --gpus 64 --gpu h100 --mfu 0.4`
  - `python scripts/estimate.py chinchilla --params 7e9` (or `--tokens 1.4e13`)
  - `python scripts/estimate.py batch --gpus 8 --micro 4 --seq 4096 --accum 8 [--target-tokens 4e6]`
  - `python scripts/estimate.py lr --params 8e9 --stage sft` (ladder lookup with sources)
- `assets/templates/` — ready-to-edit configs: TRL SFT/DPO/QLoRA YAMLs,
  Accelerate+DeepSpeed ZeRO-3 launcher, DeepSpeed ZeRO-2/ZeRO-3-offload
  JSONs, and a framework-neutral 1B pretrain spec. Hand these to the user
  instead of writing configs from scratch; adjust values per the references.
- `assets/templates/experiment_dashboard.html` — self-contained experiment
  dashboard implementing the report-detail standard (experiment-tracking.md
  §4; the §4.8 agentic-RL collapsible-trajectory rendering is the one
  extension left per-experiment): status-strip stat tiles, goal-progress
  panel (target, gap, trend ETA, on/off-track), captioned SVG line
  charts with TB/W&B-style smoothing (raw ghost) and log-y toggles, eval
  table, sweep scatter, run/experiment ladder with gates + repro metadata,
  decision ledger, inference-vs-GT sample gallery (text/audio/image), job
  chain, staleness self-check banner, provenance footer. Data lives in one
  JSON `const` at the top;
  empty sections hide themselves. Copy to `docs/index.html` in the
  experiment repo and edit only that block per run.

## Honesty rules

- Never present an unverified number as verified. The references mark each
  value `[config]` (read from an official config/model card), `[paper]`
  (stated in the paper), or `[reported]` (secondary source) — propagate
  that marking into answers when precision matters.
- When the user's setup falls outside all published regimes (exotic
  architecture, unusual modality), say so, anchor to the nearest regime,
  and recommend a small pilot run instead of false confidence.
- Cost estimates: always label assumed $/GPU-hour as an assumption.