Dlbricks Benchmark Eval

dlbricks-benchmark-eval

DLBricks: Composable Benchmark Generation to Reduce Deep Learning Benchmarking Effort on CPUs (Extended) — Li et al. (2019) (arXiv:1911.07967, 2019)

What this evaluates

Evaluates the accuracy of a composable benchmark generation framework in estimating deep learning model inference latency on CPUs, and measures the computational speedup achieved by benchmarking unique layer sequences instead of running full end-to-end models.

Datasets

• 50 DL Models — total 50; splits: test (50)

Metrics

• Normalized Latency (primary) — range: percent
  • Geometric mean of the ratio between constructed latency (estimated from layer sequences) and actual end-to-end latency. Values closer to 100% indicate higher estimation accuracy.
• Benchmarking Time Speedup — range: ratio
  • Sum of all models' actual end-to-end latencies divided by the sum of all generated layer-sequence benchmark latencies. Higher values indicate greater time savings.

Input / output format

Input: Deep learning model architecture (decomposed into layer sequences of varying granularity G) and target CPU hardware configuration.

Output: Estimated inference latency (from layer sequences) and actual end-to-end inference latency per model.

Scoring recipe

# 1. Collect latencies: run each model/benchmark 100 times, take 20% trimmed mean
actual_lat = [trimmed_mean(run(model, n=100)) for model in models]
constructed_lat = [estimate_from_layers(model, G) for model in models]

# 2. Compute Normalized Latency (Accuracy)
norm_lat = [c / a for c, a in zip(constructed_lat, actual_lat)]
accuracy = geometric_mean(norm_lat)

# 3. Compute Benchmarking Time Speedup
speedup = sum(actual_lat) / sum(constructed_lat)

Common pitfalls

• The evaluation uses a 20% trimmed mean over 100 runs rather than a simple average, which significantly affects reported latency values.
• Accuracy and speedup are highly dependent on the benchmark granularity parameter G; lower G yields higher speedup but slightly lower accuracy, creating a system-dependent trade-off.
• Results are reported on specific Amazon EC2 instances (c4/c5 series with Intel Platinum/Xeon CPUs); performance does not generalize to other CPU architectures or cloud providers without re-evaluation.

Evidence (verbatim from paper)

Each model or benchmark is run 100 times and the 20% trimmed mean is reported. The benchmarking time speedup is calculated as the sum of the end-to-end latency of all models divided by the sum of the latency of all the generated benchmarks. Figure 11 shows the geometric mean of the normalized latency (constructed vs end-to-end latency) of all the 50 models across systems and benchmark granularities.

Citation

@misc{li2019dlbricks,
  title={DLBricks: Composable Benchmark Generation to Reduce Deep Learning Benchmarking Effort on CPUs (Extended)},
  author={Li et al. (2019)},
  year={2019},
  note={arXiv:1911.07967}
}

• arXiv: 1911.07967