Mlperf Hardware Eval

mlperf-hardware-eval

Violet: Architecturally Exposed Orchestration, Movement, and Placement for Generalized Deep Learning — Davies et al. (2021) (arXiv:2112.02204, 2021)

What this evaluates

Evaluates the performance and energy efficiency of a proposed deep learning hardware architecture against state-of-the-art GPUs and specialized accelerators. It probes the architecture's ability to handle diverse DL workloads (CNNs, transformers, RNNs) across different batch sizes and its software maturity for general-purpose mapping.

Datasets

• MLPerf benchmark suite — total ?; splits: test (-1)

Metrics

• Speedup (primary) — range: other
  • Ratio of throughput measured in samples/sec between the proposed architecture and the baseline (A100). Higher values indicate better performance.
• Rel. Efficiency (primary) — range: other
  • Ratio of average energy per operation (pJ/op) between baseline and proposed architecture. Lower pJ/op is better, so efficiency = pJ_op_baseline / pJ_op_proposed.
• Compute resource utilization — range: percent
  • Percentage of available compute resources actively used during workload execution.
• Area efficiency — range: other
  • Peak compute throughput normalized by die area, measured in TOP/s/mm^2.

Input / output format

Input: MLPerf benchmark suite workloads (Resnet50, SSD-Resnet34, MaskRCNN, Bert Large 128, RNN-T, Alexnet, Mobilenet) executed at small batch size (1) and large batch size (64-128).

Output: Throughput (samples/sec), power consumption (W), energy per operation (pJ/op), compute resource utilization (%), and area efficiency (TOP/s/mm^2).

Scoring recipe

speedup = throughput_proposed / throughput_baseline
rel_efficiency = pJ_op_baseline / pJ_op_proposed
geo_mean_speedup = exp(mean(log(speedup)))
geo_mean_efficiency = exp(mean(log(rel_efficiency)))
compute_utilization = (active_cycles / total_cycles) * 100
area_efficiency = peak_tops / die_area_mm2

Common pitfalls

• Hardware simulation results may differ from real silicon due to unmodeled memory latency and mapping overheads.
• Performance is highly sensitive to batch size; small batches favor the proposed architecture while large batches narrow the gap.
• Compute utilization metrics are heavily dependent on architecture-specific compiler mapping strategies and may not reflect raw hardware capability.

Evidence (verbatim from paper)

Speedup is measured by the ratio of throughput in samples/sec, and Rel. Efficiency is measured by the ratio of average pJ/op for the network. At large batch, UPCYCLE provides modest performance improvements over A100 - geo-mean 2.2X and 2.3X for inference and training performance respectively - as well as 6X power efficiency improvement in both cases.

Citation

@misc{davies2021violet,
  title={Violet: Architecturally Exposed Orchestration, Movement, and Placement for Generalized Deep Learning},
  author={Davies et al. (2021)},
  year={2021},
  note={arXiv:2112.02204}
}

• arXiv: 2112.02204