Bootstrapper

bootstrapper

Metric BootStrapper from torchmetrics (torchmetrics.BootStrapper)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with BootStrapper, or mentions torchmetrics.BootStrapper directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics import BootStrapper

# BootStrapper(base_metric: torchmetrics.metric.Metric, num_bootstraps: int = 10, mean: bool = True, std: bool = True, quantile: Union[float, torch.Tensor, NoneType] = None, raw: bool = False, sampling_strategy: str = 'poisson', **kwargs: Any) -> None

Library docstring

Using `Turn a Metric into a Bootstrapped`_.

That can automate the process of getting confidence intervals for metric values. This wrapper
class basically keeps multiple copies of the same base metric in memory and whenever ``update`` or
``forward`` is called, all input tensors are resampled (with replacement) along the first dimension.

Args:
    base_metric: base metric class to wrap
    num_bootstraps: number of copies to make of the base metric for bootstrapping
    mean: if ``True`` return the mean of the bootstraps
    std: if ``True`` return the standard deviation of the bootstraps
    quantile: if given, returns the quantile of the bootstraps. Can only be used with pytorch version 1.6 or higher
    raw: if ``True``, return all bootstrapped values
    sampling_strategy:
        Determines how to produce bootstrapped samplings. Either ``'poisson'`` or ``multinomial``.
        If ``'possion'`` is chosen, the number of times each sample will be included in the bootstrap
        will be given by :math:`n\sim Poisson(\lambda=1)`, which approximates the true bootstrap distribution
        when the number of samples is large. If ``'multinomial'`` is chosen, we will apply true bootstrapping
        at the batch level to approximate bootstrapping over the hole dataset.
    kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info.

Example::
    >>> from pprint import pprint
    >>> from torch import randint
    >>> from torchmetrics.wrappers import BootStrapper
    >>> from torchmetrics.classification import MulticlassAccuracy
    >>> base_metric = MulticlassAccuracy(num_classes=5, average='micro')
    >>> bootstrap = BootStrapper(base_metric, num_bootstraps=20)
    >>> bootstrap.update(randint(5, (20,)), randint(5, (20,)))
    >>> output = bootstrap.compute()
    >>> pprint(output)
    {'mean': tensor(0.2089), 'std': tensor(0.0772)}

Quick recipe

import torchmetrics as _m
score = _m.BootStrapper(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).