Clipscore

clipscore

Metric CLIPScore from torchmetrics (torchmetrics.multimodal.CLIPScore)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with CLIPScore, or mentions torchmetrics.multimodal.CLIPScore directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics.multimodal import CLIPScore

# CLIPScore(model_name_or_path: Union[Literal['openai/clip-vit-base-patch16', 'openai/clip-vit-base-patch32', 'openai/clip-vit-large-patch14-336', 'openai/clip-vit-large-patch14', 'jinaai/jina-clip-v2', 'zer0int/LongCLIP-L-Diffusers', 'zer0int/LongCLIP-GmP-ViT-L-14'], Callable[[], tuple[None, None]]] = 'openai/clip-vit-large-patch14', **kwargs: Any) -> None

Library docstring

Calculates `CLIP Score`_ which is a text-to-image similarity metric.

CLIP Score is a reference free metric that can be used to evaluate the correlation between a generated caption for
an image and the actual content of the image, as well as the similarity between texts or images. It has been found
to be highly correlated with human judgement. The metric is defined as:

.. math::
    \text{CLIPScore(I, C)} = max(100 * cos(E_I, E_C), 0)

which corresponds to the cosine similarity between visual `CLIP`_ embedding :math:`E_i` for an image :math:`i` and
textual CLIP embedding :math:`E_C` for an caption :math:`C`. The score is bound between 0 and 100 and the closer
to 100 the better.

Additionally, the CLIP Score can be calculated for the same modalities:

.. math::
    \text{CLIPScore(I_1, I_2)} = max(100 * cos(E_{I_1}, E_{I_2}), 0)

where :math:`E_{I_1}` and :math:`E_{I_2}` are the visual embeddings for images :math:`I_1` and :math:`I_2`.

.. math::
    \text{CLIPScore(T_1, T_2)} = max(100 * cos(E_{T_1}, E_{T_2}), 0)

where :math:`E_{T_1}` and :math:`E_{T_2}` are the textual embeddings for texts :math:`T_1` and :math:`T_2`.

.. caution::
    Metric is not scriptable

.. note::
    The default CLIP and processor used in this implementation has a maximum sequence length of 77 for text
    inputs. If you need to process longer captions, you can use the `zer0int/LongCLIP-L-Diffusers` model which
    has a maximum sequence length of 248.

As input to ``forward`` and ``update`` the metric accepts the following input

- source: Source input.

    This can be:

    - Images: ``Tensor`` or list of ``Tensor``

        If a single tensor, it should have shape ``(N, C, H, W)``.
        If a list of tensors, each tensor should have shape ``(C, H, W)``.
        ``C`` is the number of channels, ``H`` and ``W`` are the height and width of the image.

    - Text: ``str`` or list of ``str``

        Either a single caption or a list of captions.

- target: Target input.

    This can be:

    - Images: ``Tensor`` or list of ``Tensor``

        If a single tensor, it should have shape ``(N, C, H, W)``.
        If a list of tensors, each tensor should have shape ``(C, H, W)``.
       

Quick recipe

import torchmetrics.multimodal as _m
score = _m.CLIPScore(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).