Framework Throughput Eval

framework-throughput-eval

PyTorch: An Imperative Style, High-Performance Deep Learning Library — Paszke et al. (2019) (arXiv:1912.01703, 2019)

What this evaluates

Evaluates the training throughput and execution efficiency of deep learning frameworks by measuring how quickly they process standard model architectures on a single GPU. It compares PyTorch against TensorFlow, MXNet, CNTK, Chainer, and PaddlePaddle to assess device utilization and runtime optimization.

Datasets

• AlexNet — total ?; splits: train (-1)
• VGG-19 — total ?; splits: train (-1)
• ResNet-50 — total ?; splits: train (-1)
• MobileNet — total ?; splits: train (-1)
• GNMTv2 — total ?; splits: train (-1)
• NCF — total ?; splits: train (-1)

Metrics

• Throughput (primary) — range: other
  • Number of training samples processed per second. Measured in images/s for vision models, tokens/s for GNMTv2, and samples/s for NCF.

Input / output format

Input: Model architecture definition and training data batches fed into the framework's training loop.

Output: Throughput value in images/tokens/samples per second, reported as mean ± standard deviation.

Scoring recipe

def compute_metric(predictions, gold):
    # predictions: dict mapping (framework, model) -> throughput_value
    # gold: reference framework names
    results = {}
    for (fw, model), val in predictions.items():
        results[(fw, model)] = val
    return results

Common pitfalls

• Throughput is highly dependent on hardware (CPU/GPU specs) and batch size, making cross-study comparisons difficult.
• Some frameworks lack official implementations for certain models, resulting in missing (N/A) data points.
• Measurements are taken during training, not inference, so optimization strategies differ from standard benchmarking.

Evidence (verbatim from paper)

Our results are summarized in Table 1. On all the benchmarks, the performance of PyTorch is within 17% of that of the fastest framework. We attribute this result to the fact that these tools offload most of the computation to the same version of the cuDNN and cuBLAS libraries. ... Throughput is measured in images per second for the AlexNet, VGG-19, ResNet-50, and MobileNet models, in tokens per second for the GNMTv2 model, and in samples per second for the NCF model.

Citation

@misc{paszke2019pytorch,
  title={PyTorch: An Imperative Style, High-Performance Deep Learning Library},
  author={Paszke et al. (2019)},
  year={2019},
  note={arXiv:1912.01703}
}

• arXiv: 1912.01703