Kldivergence

kldivergence

Metric KLDivergence from torchmetrics (torchmetrics.KLDivergence)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with KLDivergence, or mentions torchmetrics.KLDivergence directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics import KLDivergence

# KLDivergence(log_prob: bool = False, reduction: Literal['mean', 'sum', 'none', None] = 'mean', **kwargs: Any) -> None

Library docstring

Compute the `KL divergence`_.

.. math::
    D_{KL}(P||Q) = \sum_{x\in\mathcal{X}} P(x) \log\frac{P(x)}{Q{x}}

Where :math:`P` and :math:`Q` are probability distributions where :math:`P` usually represents a distribution
over data and :math:`Q` is often a prior or approximation of :math:`P`. It should be noted that the KL divergence
is a non-symmetrical metric i.e. :math:`D_{KL}(P||Q) \neq D_{KL}(Q||P)`.

As input to ``forward`` and ``update`` the metric accepts the following input:

- ``p`` (:class:`~torch.Tensor`): a data distribution with shape ``(N, d)``
- ``q`` (:class:`~torch.Tensor`): prior or approximate distribution with shape ``(N, d)``

As output of ``forward`` and ``compute`` the metric returns the following output:

- ``kl_divergence`` (:class:`~torch.Tensor`): A tensor with the KL divergence

Args:
    log_prob: bool indicating if input is log-probabilities or probabilities. If given as probabilities,
        will normalize to make sure the distributes sum to 1.
    reduction:
        Determines how to reduce over the ``N``/batch dimension:

        - ``'mean'`` [default]: Averages score across samples
        - ``'sum'``: Sum score across samples
        - ``'none'`` or ``None``: Returns score per sample

    kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info.

Raises:
    TypeError:
        If ``log_prob`` is not an ``bool``.
    ValueError:
        If ``reduction`` is not one of ``'mean'``, ``'sum'``, ``'none'`` or ``None``.

.. attention::
    Half precision is only support on GPU for this metric.

Example:
    >>> from torch import tensor
    >>> from torchmetrics.regression import KLDivergence
    >>> p = tensor([[0.36, 0.48, 0.16]])
    >>> q = tensor([[1/3, 1/3, 1/3]])
    >>> kl_divergence = KLDivergence()
    >>> kl_divergence(p, q)
    tensor(0.0853)

Quick recipe

import torchmetrics as _m
score = _m.KLDivergence(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).