tvm-dl-compiler-eval
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Chen et al. (2018) (arXiv:1802.04799, 2018)
What this evaluates
Evaluates the end-to-end performance and optimization capability of a deep learning compiler across diverse hardware back-ends (GPU, CPU, embedded GPU, FPGA) on standard inference workloads. It measures how effectively the compiler automatically generates high-performance kernels compared to hand-tuned vendor libraries and existing frameworks.
Datasets
- DL Inference Workloads (ResNet-18, MobileNet, LSTM, DQN, DCGAN) — total ?; splits: (unstated)
Metrics
speedup (primary) — range: other
- Calculated as the execution time of the baseline framework/library divided by the execution time of the TVM-generated kernel for the same workload and hardware configuration.
Input / output format
Input: Deep learning computational graphs with specified operator configurations (e.g., conv2d dimensions, kernel size, stride, padding) and target hardware backend specifications.
Output: Hardware-specific optimized kernel code (e.g., CUDA, C, ARM assembly) and runtime execution binaries capable of running on the target platform.
Scoring recipe
def compute_speedup(baseline_time, tvm_time):
if tvm_time <= 0:
return float('inf')
return baseline_time / tvm_time
# Evaluated per workload and hardware backend.
# Baselines include MXNet, TensorFlow, TFLite, ARM Compute Library, Caffe2.
Common pitfalls
- Comparing TVM against baselines that use heavily hand-tuned vendor libraries (e.g., cuDNN, cuBLAS) which may have operator coverage advantages not available to auto-generated compilers.
- Ignoring workload/operator support gaps in baselines (e.g., depthwise conv, low-precision ops, LSTM, DCGAN were unsupported by some baselines at the time).
- Overlooking Amdahl's law bottlenecks on hybrid systems (e.g., FPGA offloading speedup is limited by CPU-bound residual/activation layers).
Evidence (verbatim from paper)
TVM outperforms the baselines, with speedups ranging from 1.6× to 3.8× due to both joint graph optimization and the automatic optimizer, which generates high-performance fused operators.
Citation
@misc{chen2018tvm,
title={TVM: An Automated End-to-End Optimizing Compiler for Deep Learning},
author={Chen et al. (2018)},
year={2018},
note={arXiv:1802.04799}
}
1---2name: tvm-dl-compiler-eval3description: Evaluates the end-to-end performance and optimization capability of a deep learning compiler across diverse hardware back-ends (GPU, CPU, embedded GPU, FPGA) on standard inference workloads. It measures how effectively the compiler automatically generates high-performance kernels compared to hand-tuned vendor libraries and existing frameworks. Use when the user wants to benchmark on DL Inference Workloads (ResNet-18, MobileNet, LSTM, DQN, DCGAN), or asks about evaluating this task. Reports speedup.4---56# tvm-dl-compiler-eval78> TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Chen et al. (2018) (arXiv:1802.04799, 2018)910## What this evaluates1112Evaluates the end-to-end performance and optimization capability of a deep learning compiler across diverse hardware back-ends (GPU, CPU, embedded GPU, FPGA) on standard inference workloads. It measures how effectively the compiler automatically generates high-performance kernels compared to hand-tuned vendor libraries and existing frameworks.1314## Datasets1516- **DL Inference Workloads (ResNet-18, MobileNet, LSTM, DQN, DCGAN)** — total ?; splits: (unstated)1718## Metrics1920- `speedup` **(primary)** — range: other21 - Calculated as the execution time of the baseline framework/library divided by the execution time of the TVM-generated kernel for the same workload and hardware configuration.2223## Input / output format2425**Input**: Deep learning computational graphs with specified operator configurations (e.g., conv2d dimensions, kernel size, stride, padding) and target hardware backend specifications.2627**Output**: Hardware-specific optimized kernel code (e.g., CUDA, C, ARM assembly) and runtime execution binaries capable of running on the target platform.2829## Scoring recipe3031```python32def compute_speedup(baseline_time, tvm_time):33 if tvm_time <= 0:34 return float('inf')35 return baseline_time / tvm_time3637# Evaluated per workload and hardware backend.38# Baselines include MXNet, TensorFlow, TFLite, ARM Compute Library, Caffe2.39```4041## Common pitfalls4243- Comparing TVM against baselines that use heavily hand-tuned vendor libraries (e.g., cuDNN, cuBLAS) which may have operator coverage advantages not available to auto-generated compilers.44- Ignoring workload/operator support gaps in baselines (e.g., depthwise conv, low-precision ops, LSTM, DCGAN were unsupported by some baselines at the time).45- Overlooking Amdahl's law bottlenecks on hybrid systems (e.g., FPGA offloading speedup is limited by CPU-bound residual/activation layers).4647## Evidence (verbatim from paper)4849> TVM outperforms the baselines, with speedups ranging from 1.6× to 3.8× due to both joint graph optimization and the automatic optimizer, which generates high-performance fused operators.5051## Citation5253```bibtex54@misc{chen2018tvm,55 title={TVM: An Automated End-to-End Optimizing Compiler for Deep Learning},56 author={Chen et al. (2018)},57 year={2018},58 note={arXiv:1802.04799}59}60```6162- arXiv: 1802.04799
qhjqhj00