qhjqhj00 bmdal-regression-eval
A Framework and Benchmark for Deep Batch Active Learning for Regression — Holzmüller et al. (2022) (arXiv:2203.09410, 2022)
What this evaluates
Evaluates the sample efficiency and predictive accuracy of batch-mode deep active learning methods for tabular regression tasks. It probes how well different kernel-based selection strategies reduce prediction error over sequential labeling rounds compared to random sampling.
Datasets
- UCI & OpenML Tabular Regression Benchmark — total ?; splits: train (-1), val (-1), pool (-1), test (-1); repo https://github.com/dholzmueller/bmdal_reg
Metrics
RMSE(primary) — range: other- Root Mean Squared Error on the test set. The paper reports the mean of the logarithms of RMSE over 20 random seeds, 16 active learning steps, and 15 datasets. A difference of δ between two log values corresponds to a ratio e^δ ≈ 1+δ between the geometric mean errors.
Input / output format
Input: Tabular feature vectors (2 to 379 dimensions) from UCI/OpenML repositories.
Output: Continuous regression target values predicted by a 3-layer fully connected neural network.
Scoring recipe
errors = []
for rep in range(20):
for step in range(16):
preds = model.predict(test_X)
rmse = np.sqrt(np.mean((preds - test_y)**2))
errors.append(np.log(rmse))
return np.mean(errors)
Common pitfalls
- Averaging the logarithms of error metrics rather than taking the log of the average error, which changes the interpretation to geometric mean reduction.
- Comparing methods that require ensembles or dropout, as they alter the base neural network's error landscape and are not directly comparable to single-model baselines.
- Ignoring batch size sensitivity, as naive methods like MAXDIAG degrade significantly when batch size exceeds ~256.
Evidence (verbatim from paper)
Our most important metric is the RMSE, but we will also put some focus on MAXE since it can be interpreted as a measure of robustness to distribution shifts. Generally, RMSE is more affected by rare but large errors than MAE, while the quantiles and MAXE exclusively focus on rare but large errors.
Citation
@misc{holzmuller2022bmdal,
title={A Framework and Benchmark for Deep Batch Active Learning for Regression},
author={Holzmüller et al. (2022)},
year={2022},
note={arXiv:2203.09410}
}
- arXiv: 2203.09410