qhjqhj00 stochastic-ackley-eval
Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling — Jacquier et al. (2020) (arXiv:2005.13506, 2020)
What this evaluates
Evaluates the ability of uncertainty-aware deep neural networks to approximate a highly irregular, multi-extremum function and quantify predictive uncertainty. It specifically probes how well the models handle in-distribution versus out-of-distribution parameter regimes.
Datasets
- Stochastic Ackley Function — total 600; splits: train (500), test (100); repo https://github.com/pierremtb/POD-UQNN
Metrics
Relative Error (RE)(primary) — range: [0, 1] | percent- Normalized difference between predicted and true values: RE = mean(|y_pred - y_true|) / mean(|y_true|).
Mean Prediction Interval Width (MPIW)— range: [0, ∞)- Average width of the prediction intervals across the dataset, computed as the mean of (upper bound - lower bound) where bounds are mean ± 2×std.
Input / output format
Input: Normalized non-spatial parameters X (size P=3) and 2D spatial coordinates (x,y) over a 400×400 grid (160,000 DOFs).
Output: Predicted mean ū_D^μ and predicted standard deviation ū_D^σ for the function value u.
Scoring recipe
def compute_metrics(y_true, y_pred_mean, y_pred_std):
re = np.mean(np.abs(y_pred_mean - y_true)) / np.mean(np.abs(y_true))
interval_width = 4 * y_pred_std # ±2 std
mpiw = np.mean(interval_width)
return re, mpiw
Common pitfalls
- The dataset is synthetically generated via Latin Hypercube Sampling, not a standard public benchmark.
- Validation split is fixed at 20% of the generated dataset, with manual early stopping rather than automated.
- Out-of-distribution evaluation uses a specific disjoint parameter range Ω_out = [-2,-1] ∪ [1,2].
Evidence (verbatim from paper)
The relative error RE defined in (21) is computed at each training epoch for both the training set and the validation set. ... To picture the difference in confidence between in- and out-of-scope predictions quantitatively, we computed MPIW_tst=0.15 and MPIW_out=10.0.
Citation
@misc{jacquier2020poduqnn,
title={Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling},
author={Jacquier et al. (2020)},
year={2020},
note={arXiv:2005.13506}
}
- arXiv: 2005.13506