Mlperf Bert Eval

mlperf-bert-eval

Boosting Distributed Training Performance of the Unpadded BERT Model — Zeng et al. (2022) (arXiv:2208.08124, 2022)

What this evaluates

Evaluates the distributed training efficiency of the BERT-Large model on variable-length masked language modeling tasks. It measures how quickly the model converges to a target accuracy and the sustained throughput achieved across multiple GPUs.

Datasets

• MLPerf BERT — total ?; splits: train (-1), eval (-1)

Metrics

• Time to 72% MLM accuracy (primary) — range: minutes
  • Wall-clock time in minutes required for the model to reach 72% Masked Language Modeling accuracy on the held-out evaluation dataset during pre-training.
• Throughput (Samples/s) — range: samples/s
  • Number of training samples processed per second, calculated as total batch size divided by training time.

Input / output format

Input: Variable-length tokenized sequences for Masked Language Modeling (MLM) pre-training, processed without padding overhead.

Output: Aggregated performance metrics: convergence time in minutes and training throughput in samples per second.

Scoring recipe

def compute_throughput(total_samples, total_time_seconds):
    return total_samples / total_time_seconds

def compute_convergence_time(eval_dataset, model, target_acc=0.72):
    start_time = current_time()
    for batch in eval_dataset:
        acc = evaluate_mlm_accuracy(model, batch)
        if acc >= target_acc:
            return current_time() - start_time
    return float('inf')

Common pitfalls

• Hardware is strictly fixed to 8x NVIDIA A100 400W GPUs; scaling to different GPU counts or memory configurations will yield different speedups.
• Global batch size is fixed at 448 (56 per GPU); changing it alters both throughput and convergence dynamics.
• Unpadded computation eliminates padding token overhead, making direct comparisons with padded baselines (e.g., DeepSpeed) misleading without normalizing for effective sequence length.

Evidence (verbatim from paper)

As shown in Table [III], our work can achieve the fastest throughput with roughly 2578 Samples/s. The metrics for the MLPerf BERT model is the time to train that converges to 72% MLM (Masked Language Modeling) accuracy on the evaluation data set.

Citation

@misc{zeng2022boosting,
  title={Boosting Distributed Training Performance of the Unpadded BERT Model},
  author={Zeng et al. (2022)},
  year={2022},
  note={arXiv:2208.08124}
}

• arXiv: 2208.08124