Multiclasscalibrationerror

multiclasscalibrationerror

Metric MulticlassCalibrationError from torchmetrics (torchmetrics.classification.MulticlassCalibrationError)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with MulticlassCalibrationError, or mentions torchmetrics.classification.MulticlassCalibrationError directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics.classification import MulticlassCalibrationError

# MulticlassCalibrationError(num_classes: int, n_bins: int = 15, norm: Literal['l1', 'l2', 'max'] = 'l1', ignore_index: Optional[int] = None, validate_args: bool = True, **kwargs: Any) -> None

Library docstring

`Top-label Calibration Error`_ for multiclass tasks.

The expected calibration error can be used to quantify how well a given model is calibrated e.g. how well the
predicted output probabilities of the model matches the actual probabilities of the ground truth distribution.
Three different norms are implemented, each corresponding to variations on the calibration error metric.

.. math::
    \text{ECE} = \sum_i^N b_i \|(p_i - c_i)\|, \text{L1 norm (Expected Calibration Error)}

.. math::
    \text{MCE} =  \max_{i} (p_i - c_i), \text{Infinity norm (Maximum Calibration Error)}

.. math::
    \text{RMSCE} = \sqrt{\sum_i^N b_i(p_i - c_i)^2}, \text{L2 norm (Root Mean Square Calibration Error)}

Where :math:`p_i` is the top-1 prediction accuracy in bin :math:`i`, :math:`c_i` is the average confidence of
predictions in bin :math:`i`, and :math:`b_i` is the fraction of data points in bin :math:`i`. Bins are constructed
in an uniform way in the [0,1] range.

As input to ``forward`` and ``update`` the metric accepts the following input:

- ``preds`` (:class:`~torch.Tensor`): A float tensor of shape ``(N, C, ...)`` containing probabilities or logits for
  each observation. If preds has values outside [0,1] range we consider the input to be logits and will auto apply
  softmax per sample.
- ``target`` (:class:`~torch.Tensor`): An int tensor of shape ``(N, ...)`` containing ground truth labels, and
  therefore only contain values in the [0, n_classes-1] range (except if `ignore_index` is specified).

.. tip::
   Additional dimension ``...`` will be flattened into the batch dimension.

As output to ``forward`` and ``compute`` the metric returns the following output:

- ``mcce`` (:class:`~torch.Tensor`): A scalar tensor containing the calibration error

Args:
    num_classes: Integer specifying the number of classes
    n_bins: Number of bins to use when computing the metric.
    norm: Norm used to compare empirical and expected probability bins.
    ignore_index:
        Specifies a target value that is ignored and does not contribute to the metric calculation
    validate_args: bool indicating if input arguments and tensors should be validated for correctness.
        Set to

Quick recipe

import torchmetrics.classification as _m
score = _m.MulticlassCalibrationError(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).