Uncertainty Robustness Eval

uncertainty-robustness-eval

Maximum Likelihood Uncertainty Estimation: Robustness to Outliers — Nair et al. (2022) (arXiv:2202.03870, 2022)

What this evaluates

Benchmarks the robustness of uncertainty estimation methods against label outliers and distribution shifts. It evaluates whether predicted prediction intervals and uncertainty quantifications maintain calibration and accuracy when training data is contaminated with noise or adversarial perturbations.

Datasets

• Synthetic 1D regression dataset — total ?; splits: train (-1), test (-1)
• Real-world regression datasets — total ?; splits: train (-1), test (-1)
• NYU-Depth-v2 — total 27000; splits: train (27000), test (-1)

Metrics

• Interval score (primary) — range: other
  • Calculates the quality of a prediction interval [l, u] for a target y at confidence level 1-alpha: S_alpha^int(l,u;y) = (u-l) + (2/alpha)*(l-y)*1_{y<l} + (2/alpha)*(y-u)*1_{y>u}. Rewards narrow intervals and penalizes misses. Alpha is fixed to 0.05 (95% interval).
• RMSE — range: other
  • Root Mean Square Error between predicted mean and true target. Standard regression accuracy metric.

Input / output format

Input: Regression: scalar input x and target y. Depth estimation: RGB image of shape (160, 128, 3) and corresponding depth map of shape (160, 128).

Output: Predicted distribution parameters (mean μ and variance σ²), from which the 95% prediction interval [l, u] is derived.

Scoring recipe

def interval_score(y_true, l, u, alpha=0.05):
    miss_penalty = (2 / alpha) * np.maximum(l - y_true, 0) + \
                   (2 / alpha) * np.maximum(y_true - u, 0)
    score = (u - l) + miss_penalty
    return np.mean(score)

Common pitfalls

• Using Negative Log-Likelihood (NLL) to compare uncertainty across different distributional assumptions (e.g., Gaussian vs Laplace), as NLL is not comparable across distributions.
• Equating prediction accuracy (RMSE) with uncertainty quality; the paper shows breakaway points for output prediction and uncertainty prediction differ significantly across methods.

Evidence (verbatim from paper)

Most of the literature in uncertainty estimation for regression use root mean square error (RMSE) for comparing performance of model and NLL for comparing performance of uncertainty. Although NLL is a proper Scoring Rule it is not comparable across different distributions. In this work we use Interval score Scoring Rule. The Interval score is defined as : S_{\alpha}^{int}(l,u;y)=(u-l)+\frac{2}{\alpha}(l-y)\mathbf{1}{y<l}+\frac{2}{\alpha}(y-u)\mathbf{1}{y>u}

Citation

@misc{nair2022maximum,
  title={Maximum Likelihood Uncertainty Estimation: Robustness to Outliers},
  author={Nair et al. (2022)},
  year={2022},
  note={arXiv:2202.03870}
}

• arXiv: 2202.03870