Uncertainty

uncertainty

The Gaia-ESO Survey: Chemical evolution of Mg and Al in the Milky Way with Machine-Learning — Ambrosch et al. (2022) (arXiv:2208.08872, 2022)

What this evaluates

Evaluates a CNN's ability to predict stellar atmospheric parameters and chemical abundances from low-resolution spectra, measuring both internal consistency across model runs and agreement with established spectroscopic pipeline measurements.

Datasets

• Gaia-ESO Survey (GES) — total 30000; splits: train (-1), test (-1), observed (-1)

Metrics

• Uncertainty (primary) — range: K or dex
  • Standard deviation of predictions across eight independently trained CNN models with different random weight initializations.
• Bias & Scatter — range: K or dex
  • Mean difference and standard deviation around the 1:1 relation between CNN predictions and Gaia-ESO pipeline measurements.

Input / output format

Input: GIRAFFE low-resolution stellar spectra (flux as a function of wavelength).

Output: Five continuous values: T_eff (K), log g (dex), [Mg/Fe] (dex), [Al/Fe] (dex), [Fe/H] (dex).

Scoring recipe

# preds: list of 8 model predictions per star
# gold: GES pipeline measurements
internal_uncertainty = np.std(preds, axis=0)
bias = np.mean(preds[0] - gold)
scatter = np.std(preds[0] - gold)

Common pitfalls

• Internal uncertainty measures model initialization variance, not data noise or true prediction error.
• Performance degrades significantly for low S/N spectra (<30) and extreme abundance values due to limited training data.
• Strong label correlations in training data can cause the network to learn spurious correlations instead of physical spectral features.

Evidence (verbatim from paper)

We define the uncertainties of our results as the dispersion between the label predictions from the eight CNN models. In Fig. 8 we display the distribution of the label uncertainties σ(Label) relative to the predicted label values of our five labels. ... The mean uncertainties of the label predictions are small: 24K for T_eff, 0.03 for log(g), 0.02 dex for [Mg/Fe], 0.03 dex for [Al/Fe], and 0.02 dex for [Fe/H].

Citation

@misc{ambrosch2022gaia,
  title={The Gaia-ESO Survey: Chemical evolution of Mg and Al in the Milky Way with Machine-Learning},
  author={Ambrosch et al. (2022)},
  year={2022},
  note={arXiv:2208.08872}
}

• arXiv: 2208.08872