Paradnn Hardware Bench

paradnn-hardware-bench

Benchmarking TPU, GPU, and CPU Platforms for Deep Learning — Wang et al. (2019) (arXiv:1907.10701, 2019)

What this evaluates

Evaluates how different deep learning model architectures and hyperparameters affect hardware performance across TPU, GPU, and CPU platforms. It probes the interaction between model attributes (size, type, batch size) and hardware bottlenecks like memory bandwidth, compute utilization, and data infeed overhead.

Datasets

• ParaDnn — total ?; splits: sweep (-1)

Metrics

• performance (primary) — range: other
  • Measured as hardware throughput (e.g., operations/sec or images/sec) and FLOPS utilization across varying model configurations. Evaluated at the largest batch size supported by each hardware platform.

Input / output format

Input: Parameterized end-to-end DNN models (FC, CNN, RNN) with variable hyperparameters including number of layers, nodes/filters, input/output dimensions, sequence lengths, vocabulary sizes, and training batch sizes.

Output: Performance metrics (throughput, FLOPS utilization, memory bandwidth usage) per hardware platform (TPU, GPU, CPU) for each model configuration.

Scoring recipe

for config in ParaDnn_sweep:
    batch = max_batch_size(hardware, config)
    run_model(config, batch, hardware)
    record_throughput(hardware, config)
    record_flops_utilization(hardware, config)
compare_platforms(throughput, flops_utilization)

Common pitfalls

• Assuming fixed model sizes; ParaDnn explicitly sweeps hyperparameters across wide ranges.
• Ignoring data infeed overhead and memory bandwidth constraints, which heavily limit TPU performance on certain workloads.
• Using batch sizes beyond hardware limits, which causes out-of-memory errors or degraded performance.

Evidence (verbatim from paper)

Their limitation is that they only contain today’s deep learning models, which may become obsolete as DL models evolve rapidly. Further, they fail to reveal deep insights into interactions between DL model attributes and hardware performance, since the benchmarks are sparse points in the vast space of deep learning models.

Citation

@misc{wang2019paradnn,
  title={Benchmarking TPU, GPU, and CPU Platforms for Deep Learning},
  author={Wang et al. (2019)},
  year={2019},
  note={arXiv:1907.10701}
}

• arXiv: 1907.10701