qhjqhj00 training_throughput
Pathways: Asynchronous Distributed Dataflow for ML — Barham et al. (2022) (arXiv:2203.12533, 2022)
What this evaluates
Evaluates the scalability and efficiency of distributed machine learning systems by measuring how many training tokens each system can process per second across different hardware topologies and model parallelism strategies.
Datasets
- T5 model configurations (Raffel et al. 2019) — total ?; splits: train (-1)
- 3B Decoder-only Transformer — total ?; splits: train (-1)
- 64B/136B Decoder-only Transformer — total ?; splits: train (-1)
Metrics
training throughput (tokens/second)(primary) — range: tokens/s- Total number of training tokens processed divided by the wall-clock time required for training steps. Reported in thousands of tokens per second (k tokens/s).
computation throughput (computations/second)— range: computations/s- Number of micro-benchmark computations executed per second on accelerators.
Input / output format
Input: Distributed training code (JAX, TensorFlow, Pathways, or Ray) and fixed global batch sizes fed to TPU or GPU clusters.
Output: Throughput values (tokens/second or computations/second) reported in tables and figures.
Scoring recipe
def compute_throughput(total_units, wall_clock_seconds):
return total_units / wall_clock_seconds
# total_units is either training tokens or micro-benchmark computations.
# Values are typically reported in k tokens/s or computations/s.
Common pitfalls
- Throughput is highly sensitive to hardware interconnects (ICI vs DCN) and TPU generations; cross-system comparisons require identical hardware configurations.
- Batch size and micro-batch configuration must be kept constant when comparing SPMD vs. pipeline parallelism, as they directly affect throughput.
- Numerical correctness (identical perplexity) is verified but not reported as a quantitative metric, so performance differences are purely system-level.
Evidence (verbatim from paper)
Table 1 shows the training throughput (tokens/second) for Text-to-text Transformer model with various model sizes (up to 11 billion parameters), training on different number of accelerators. As expected, since the model code is the same, the models trained on JAX and Pathways achieve the same perplexity in the same number of steps. Over all tested model sizes, the two systems show identical performance since realistic computations are large enough to mask single-controller overheads.
Citation
@misc{barham2022pathways,
title={Pathways: Asynchronous Distributed Dataflow for ML},
author={Barham et al. (2022)},
year={2022},
note={arXiv:2203.12533}
}
- arXiv: 2203.12533