Cohen Kappa Score

cohen-kappa-score

Metric cohen_kappa_score from scikit-learn (sklearn.metrics.cohen_kappa_score)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with cohen_kappa_score, or mentions sklearn.metrics.cohen_kappa_score directly, or wants the standard scikit-learn implementation.

Reference signature

from sklearn.metrics import cohen_kappa_score

# cohen_kappa_score(y1, y2, *, labels=None, weights=None, sample_weight=None)

Library docstring

Compute Cohen's kappa: a statistic that measures inter-annotator agreement.

This function computes Cohen's kappa [1]_, a score that expresses the level
of agreement between two annotators on a classification problem. It is
defined as

.. math::
    \kappa = (p_o - p_e) / (1 - p_e)

where :math:`p_o` is the empirical probability of agreement on the label
assigned to any sample (the observed agreement ratio), and :math:`p_e` is
the expected agreement when both annotators assign labels randomly.
:math:`p_e` is estimated using a per-annotator empirical prior over the
class labels [2]_.

Read more in the :ref:`User Guide <cohen_kappa>`.

Parameters
----------
y1 : array-like of shape (n_samples,)
    Labels assigned by the first annotator.

y2 : array-like of shape (n_samples,)
    Labels assigned by the second annotator. The kappa statistic is
    symmetric, so swapping ``y1`` and ``y2`` doesn't change the value.

labels : array-like of shape (n_classes,), default=None
    List of labels to index the matrix. This may be used to select a
    subset of labels. If `None`, all labels that appear at least once in
    ``y1`` or ``y2`` are used. Note that at least one label in `labels` must be
    present in `y1`, even though this function is otherwise agnostic to the order
    of `y1` and `y2`.

weights : {'linear', 'quadratic'}, default=None
    Weighting type to calculate the score. `None` means not weighted;
    "linear" means linear weighting; "quadratic" means quadratic weighting.

sample_weight : array-like of shape (n_samples,), default=None
    Sample weights.

Returns
-------
kappa : float
    The kappa statistic, which is a number between -1 and 1. The maximum
    value means complete agreement; zero or lower means chance agreement.

References
----------
.. [1] :doi:`J. Cohen (1960). "A coefficient of agreement for nominal scales".
       Educational and Psychological Measurement 20(1):37-46.
       <10.1177/001316446002000104>`
.. [2] `R. Artstein and M. Poesio (2008). "Inter-coder agreement for
       computational linguistics". Computational Linguistics 34(4):555-596
       <https://www.mitpressjournals.org/doi/pdf/10.1162/coli.07-034-R2>`_.
.. [3] `Wikiped

Quick recipe

import sklearn.metrics as _m
score = _m.cohen_kappa_score(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).