Wa Hls4ml Eval

wa-hls4ml-eval

wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation — Hawks et al. (2025) (arXiv:2511.05615, 2025)

What this evaluates

Probes the ability of surrogate models to accurately predict FPGA resource usage (e.g., LUTs, BRAMs) and inference latency (clock cycles) for neural networks synthesized via hls4ml. It evaluates how well learned models approximate time-consuming hardware synthesis processes without running the full compilation pipeline.

Datasets

• wa-hls4ml benchmark — total 680000; splits: test (-1); repo https://github.com/fastmachinelearning/wa-hls4ml-paper

Metrics

• R² (primary) — range: (-∞, 1]
  • Coefficient of determination: R² = 1 - Σ(y_i - ŷ_i)² / Σ(y_i - ȳ)². Measures how well the predictor captures variability in the ground truth data.
• SMAPE — range: percent
  • Symmetric mean absolute percentage error: SMAPE = (200%/n) Σ |y_i - ŷ_i| / (|y_i| + |ŷ_i| + 1). Measures relative accuracy across different scales, with ε=1 added to the denominator to prevent division by zero.
• RMSE — range: absolute units
  • Root mean square error: RMSE = √(1/n Σ(y_i - ŷ_i)²). Measures the magnitude of prediction error, sensitive to larger outliers.

Input / output format

Input: Neural network architecture specifications (layer types, dimensions, precision settings), hls4ml configuration parameters, and target FPGA board details.

Output: Predicted numerical values for each FPGA resource metric and latency (clock cycles), accompanied by box plots of relative percentage errors and a detailed report describing model architecture, hyperparameters, and hardware specs.

Scoring recipe

def compute_metrics(y_true, y_pred):
    n = len(y_true)
    y_mean = sum(y_true) / n
    r2 = 1 - sum((y - yp)**2 for y, yp in zip(y_true, y_pred)) / sum((y - y_mean)**2 for y in y_true)
    smape = (200.0 / n) * sum(abs(y - yp) / (abs(y) + abs(yp) + 1) for y, yp in zip(y_true, y_pred))
    rmse = (sum((y - yp)**2 for y, yp in zip(y_true, y_pred)) / n) ** 0.5
    return {'R2': r2, 'SMAPE': smape, 'RMSE': rmse}

Common pitfalls

• Reporting resource usage as percentage utilization instead of absolute counts (LUTs, FFs, etc.).
• Omitting the ε=1 term in the SMAPE denominator, causing division-by-zero errors for zero-latency/resource predictions.
• Aggregating metrics across different variables instead of computing them separately per regression target.

Evidence (verbatim from paper)

In the above equations, y_i represents the ground truth, ŷ_i is the predicted value, and ȳ is the mean of the ground truth values. The R² score evaluates the general performance of a predictor, measuring how well it captures the variability in the data. SMAPE offers insight into the relative accuracy of the predictions and is particularly useful when comparing errors across different scales. RMSE measures the magnitude of the prediction error, with sensitivity to larger outliers.

Citation

@misc{hawks2025wahls4ml,
  title={wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation},
  author={Hawks et al. (2025)},
  year={2025},
  note={arXiv:2511.05615}
}

• arXiv: 2511.05615