Calibrationerror

calibrationerror

Metric CalibrationError from torchmetrics (torchmetrics.CalibrationError)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with CalibrationError, or mentions torchmetrics.CalibrationError directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics import CalibrationError

# CalibrationError(task: Literal['binary', 'multiclass'], n_bins: int = 15, norm: Literal['l1', 'l2', 'max'] = 'l1', num_classes: Optional[int] = None, ignore_index: Optional[int] = None, validate_args: bool = True, **kwargs: Any) -> torchmetrics.metric.Metric

Library docstring

`Top-label Calibration Error`_.

The expected calibration error can be used to quantify how well a given model is calibrated e.g. how well the
predicted output probabilities of the model matches the actual probabilities of the ground truth distribution.
Three different norms are implemented, each corresponding to variations on the calibration error metric.

.. math::
    \text{ECE} = \sum_i^N b_i \|(p_i - c_i)\|, \text{L1 norm (Expected Calibration Error)}

.. math::
    \text{MCE} =  \max_{i} (p_i - c_i), \text{Infinity norm (Maximum Calibration Error)}

.. math::
    \text{RMSCE} = \sqrt{\sum_i^N b_i(p_i - c_i)^2}, \text{L2 norm (Root Mean Square Calibration Error)}

Where :math:`p_i` is the top-1 prediction accuracy in bin :math:`i`, :math:`c_i` is the average confidence of
predictions in bin :math:`i`, and :math:`b_i` is the fraction of data points in bin :math:`i`. Bins are constructed
in an uniform way in the [0,1] range.

This function is a simple wrapper to get the task specific versions of this metric, which is done by setting the
``task`` argument to either ``'binary'`` or ``'multiclass'``. See the documentation of
:class:`~torchmetrics.classification.BinaryCalibrationError` and
:class:`~torchmetrics.classification.MulticlassCalibrationError` for the specific details of each argument influence
and examples.

Quick recipe

import torchmetrics as _m
score = _m.CalibrationError(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).