Binarycalibrationerror

binarycalibrationerror

Metric BinaryCalibrationError from torchmetrics (torchmetrics.classification.BinaryCalibrationError)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with BinaryCalibrationError, or mentions torchmetrics.classification.BinaryCalibrationError directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics.classification import BinaryCalibrationError

# BinaryCalibrationError(n_bins: int = 15, norm: Literal['l1', 'l2', 'max'] = 'l1', ignore_index: Optional[int] = None, validate_args: bool = True, **kwargs: Any) -> None

Library docstring

`Top-label Calibration Error`_ for binary tasks.

The expected calibration error can be used to quantify how well a given model is calibrated e.g. how well the
predicted output probabilities of the model matches the actual probabilities of the ground truth distribution.
Three different norms are implemented, each corresponding to variations on the calibration error metric.

.. math::
    \text{ECE} = \sum_i^N b_i \|(p_i - c_i)\|, \text{L1 norm (Expected Calibration Error)}

.. math::
    \text{MCE} =  \max_{i} (p_i - c_i), \text{Infinity norm (Maximum Calibration Error)}

.. math::
    \text{RMSCE} = \sqrt{\sum_i^N b_i(p_i - c_i)^2}, \text{L2 norm (Root Mean Square Calibration Error)}

Where :math:`p_i` is the top-1 prediction accuracy in bin :math:`i`, :math:`c_i` is the average confidence of
predictions in bin :math:`i`, and :math:`b_i` is the fraction of data points in bin :math:`i`. Bins are constructed
in an uniform way in the [0,1] range.

As input to ``forward`` and ``update`` the metric accepts the following input:

- ``preds`` (:class:`~torch.Tensor`): A float tensor of shape ``(N, ...)`` containing probabilities or logits for
  each observation. If preds has values outside [0,1] range we consider the input to be logits and will auto apply
  sigmoid per element.
- ``target`` (:class:`~torch.Tensor`): An int tensor of shape ``(N, ...)`` containing ground truth labels, and
  therefore only contain {0,1} values (except if `ignore_index` is specified). The value 1 always encodes the
  positive class.

As output to ``forward`` and ``compute`` the metric returns the following output:

- ``bce`` (:class:`~torch.Tensor`): A scalar tensor containing the calibration error

Additional dimension ``...`` will be flattened into the batch dimension.

Args:
    n_bins: Number of bins to use when computing the metric.
    norm: Norm used to compare empirical and expected probability bins.
    ignore_index:
        Specifies a target value that is ignored and does not contribute to the metric calculation
    validate_args: bool indicating if input arguments and tensors should be validated for correctness.
        Set to ``False`` for faster computations.
    kwargs: Addi

Quick recipe

import torchmetrics.classification as _m
score = _m.BinaryCalibrationError(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).