Habitat Predictor Eval

habitat-predictor-eval

A Runtime-Based Computational Performance Predictor for Deep Neural Network Training — Yu et al. (2021) (arXiv:2102.00527, 2021)

What this evaluates

Probes the ability of a runtime-based predictor to accurately estimate GPU training iteration execution times and cost-normalized throughput across different DNN architectures and GPU generations without requiring full training runs.

Datasets

• ImageNet — total ?; splits: test (-1)
• WMT'16 — total ?; splits: test (-1)
• LSUN — total ?; splits: test (-1)

Metrics

• average prediction error (primary) — range: percent
  • Calculated as the absolute difference between predicted and actual iteration execution time divided by the actual time, averaged across all GPU pairs and models, then multiplied by 100.

Input / output format

Input: Model architecture, batch size, source GPU hardware specifications, and target GPU hardware specifications.

Output: Predicted training iteration execution time (seconds), training throughput (samples/sec), and cost-normalized throughput (samples/dollar).

Scoring recipe

total_error = 0.0
count = 0
for pred, actual in zip(predictions, ground_truth):
    if actual > 0:
        total_error += abs(pred - actual) / actual
        count += 1
mape = (total_error / count) * 100
return mape

Common pitfalls

• Uses synthetic data sampled from a normal distribution rather than real dataset values, meaning I/O and data loading overheads are excluded from the measured execution time.
• Averages prediction error across all GPU pairs and models, which can obscure high errors on specific operation types or less common GPU architectures.
• Cost-normalized throughput relies on static rental prices from a specific cloud provider and time period, which may not reflect current market rates or on-premise electricity costs.

Evidence (verbatim from paper)

The average prediction error across all GPUs and models is 11.8%.

Citation

@misc{yu2021habitat,
  title={A Runtime-Based Computational Performance Predictor for Deep Neural Network Training},
  author={Yu et al. (2021)},
  year={2021},
  note={arXiv:2102.00527}
}

• arXiv: 2102.00527