Fleisskappa

fleisskappa

Metric FleissKappa from torchmetrics (torchmetrics.FleissKappa)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with FleissKappa, or mentions torchmetrics.FleissKappa directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics import FleissKappa

# FleissKappa(mode: Literal['counts', 'probs'] = 'counts', **kwargs: Any) -> None

Library docstring

Calculatees `Fleiss kappa`_ a statistical measure for inter agreement between raters.

.. math::
    \kappa = \frac{\bar{p} - \bar{p_e}}{1 - \bar{p_e}}

where :math:`\bar{p}` is the mean of the agreement probability over all raters and :math:`\bar{p_e}` is the mean
agreement probability over all raters if they were randomly assigned. If the raters are in complete agreement then
the score 1 is returned, if there is no agreement among the raters (other than what would be expected by chance)
then a score smaller than 0 is returned.

As input to ``forward`` and ``update`` the metric accepts the following input:

- ``ratings`` (:class:`~torch.Tensor`): Ratings of shape ``[n_samples, n_categories]`` or
  ``[n_samples, n_categories, n_raters]`` depedenent on ``mode``. If ``mode`` is ``counts``, ``ratings`` must be
  integer and contain the number of raters that chose each category. If ``mode`` is ``probs``, ``ratings`` must be
  floating point and contain the probability/logits that each rater chose each category.

As output of ``forward`` and ``compute`` the metric returns the following output:

- ``fleiss_k`` (:class:`~torch.Tensor`): A float scalar tensor with the calculated Fleiss' kappa score.

Args:
    mode: Whether `ratings` will be provided as counts or probabilities.
    kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info.

Example:
    >>> # Ratings are provided as counts
    >>> from torch import randint
    >>> from torchmetrics.nominal import FleissKappa
    >>> ratings = randint(0, 10, size=(100, 5)).long()  # 100 samples, 5 categories, 10 raters
    >>> metric = FleissKappa(mode='counts')
    >>> metric(ratings)
    tensor(0.0089)

Example:
    >>> # Ratings are provided as probabilities
    >>> from torch import randn
    >>> from torchmetrics.nominal import FleissKappa
    >>> ratings = randn(100, 5, 10).softmax(dim=1)  # 100 samples, 5 categories, 10 raters
    >>> metric = FleissKappa(mode='probs')
    >>> metric(ratings)
    tensor(-0.0075)

Quick recipe

import torchmetrics as _m
score = _m.FleissKappa(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).