Clipimagequalityassessment

clipimagequalityassessment

Metric CLIPImageQualityAssessment from torchmetrics (torchmetrics.multimodal.CLIPImageQualityAssessment)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with CLIPImageQualityAssessment, or mentions torchmetrics.multimodal.CLIPImageQualityAssessment directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics.multimodal import CLIPImageQualityAssessment

# CLIPImageQualityAssessment(model_name_or_path: Literal['clip_iqa', 'openai/clip-vit-base-patch16', 'openai/clip-vit-base-patch32', 'openai/clip-vit-large-patch14-336', 'openai/clip-vit-large-patch14'] = 'clip_iqa', data_range: float = 1.0, prompts: tuple[typing.Union[str, tuple[str, str]], ...] = ('quality',), **kwargs: Any) -> None

Library docstring

Calculates `CLIP-IQA`_, that can be used to measure the visual content of images.

The metric is based on the `CLIP`_ model, which is a neural network trained on a variety of (image, text) pairs to
be able to generate a vector representation of the image and the text that is similar if the image and text are
semantically similar.

The metric works by calculating the cosine similarity between user provided images and pre-defined prompts. The
prompts always comes in pairs of "positive" and "negative" such as "Good photo." and "Bad photo.". By calculating
the similartity between image embeddings and both the "positive" and "negative" prompt, the metric can determine
which prompt the image is more similar to. The metric then returns the probability that the image is more similar
to the first prompt than the second prompt.

Build in prompts are:
    * quality: "Good photo." vs "Bad photo."
    * brightness: "Bright photo." vs "Dark photo."
    * noisiness: "Clean photo." vs "Noisy photo."
    * colorfullness: "Colorful photo." vs "Dull photo."
    * sharpness: "Sharp photo." vs "Blurry photo."
    * contrast: "High contrast photo." vs "Low contrast photo."
    * complexity: "Complex photo." vs "Simple photo."
    * natural: "Natural photo." vs "Synthetic photo."
    * happy: "Happy photo." vs "Sad photo."
    * scary: "Scary photo." vs "Peaceful photo."
    * new: "New photo." vs "Old photo."
    * warm: "Warm photo." vs "Cold photo."
    * real: "Real photo." vs "Abstract photo."
    * beautiful: "Beautiful photo." vs "Ugly photo."
    * lonely: "Lonely photo." vs "Sociable photo."
    * relaxing: "Relaxing photo." vs "Stressful photo."

As input to ``forward`` and ``update`` the metric accepts the following input

- ``images`` (:class:`~torch.Tensor`): tensor with images feed to the feature extractor with shape ``(N,C,H,W)``

As output of `forward` and `compute` the metric returns the following output

- ``clip_iqa`` (:class:`~torch.Tensor` or dict of tensors): tensor with the CLIP-IQA score. If a single prompt is
  provided, a single tensor with shape ``(N,)`` is returned. If a list of prompts is provided, a dict of tensors
  is returned with the prompt as key and

Quick recipe

import torchmetrics.multimodal as _m
score = _m.CLIPImageQualityAssessment(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).