LLM Pretrain Finetune Eval

llm-pretrain-finetune-eval

Wavelet Meets Adam: Compressing Gradients for Memory-Efficient Training — Wen et al. (2025) (arXiv:2501.07237, 2025)

What this evaluates

Evaluates memory-efficient gradient compression optimizers against full-rank baselines during LLM pre-training and fine-tuning, measuring final model quality, convergence speed, memory footprint, and training throughput.

Datasets

• C4 — total ?; splits: val (-1)
• MMLU — total ?; splits: test (-1); HF cais/mmlu
• GLUE — total ?; splits: test (-1); HF glue

Metrics

• Validation PPL (primary) — range: other
  • Perplexity computed on the validation set during pre-training; lower values indicate better language modeling performance.
• Accuracy (primary) — range: percent
  • Percentage of correctly predicted answers on MMLU and GLUE tasks; averaged across sub-tasks for MMLU.
• Memory Usage — range: other
  • Estimated peak optimizer memory footprint in gigabytes (GB) during training.
• Training Throughput — range: other
  • Number of training tokens processed per second per GPU.

Input / output format

Input: Tokenized text sequences for pre-training; multiple-choice question-answer pairs for MMLU; task-specific prompts for GLUE.

Output: Next-token probability distributions for pre-training; discrete class labels for MMLU/GLUE fine-tuning.

Scoring recipe

PPL: exp(-mean(log_softmax(logits)))
Accuracy: sum(predictions == gold_labels) / total_samples * 100
Memory: peak GPU memory allocated for optimizer states
Throughput: total_tokens / (training_time * num_gpus)

Common pitfalls

• Memory estimates are approximations and depend heavily on framework implementation and optimizer state tracking.
• Throughput measurements are hardware-specific (RTX 3090 used) and may not generalize across GPU architectures.
• Fair comparison requires sweeping learning rates for each optimizer due to differing sensitivities to hyperparameters.

Evidence (verbatim from paper)

The final validation perplexity (PPL) and estimated memory usage for all methods are summarized in Table [1]. Experimental results show that GWT consistently achieves lower validation PPL while reducing memory usage compared to other memory-efficient baselines.

Citation

@misc{wen2025wavelet,
  title={Wavelet Meets Adam: Compressing Gradients for Memory-Efficient Training},
  author={Wen et al. (2025)},
  year={2025},
  note={arXiv:2501.07237}
}

• arXiv: 2501.07237