Bertscore

bertscore

Metric BERTScore from torchmetrics (torchmetrics.text.BERTScore)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with BERTScore, or mentions torchmetrics.text.BERTScore directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics.text import BERTScore

# BERTScore(model_name_or_path: Optional[str] = None, num_layers: Optional[int] = None, all_layers: bool = False, model: Optional[torch.nn.modules.module.Module] = None, user_tokenizer: Optional[Any] = None, user_forward_fn: Optional[Callable[[torch.nn.modules.module.Module, dict[str, torch.Tensor]], torch.Tensor]] = None, verbose: bool = False, idf: bool = False, device: Union[str, torch.device, NoneType] = None, max_length: int = 512, batch_size: int = 64, num_threads: int = 0, return_hash: bool = False, lang: str = 'en', rescale_with_baseline: bool = False, baseline_path: Optional[str] = None, baseline_url: Optional[str] = None, truncation: bool = False, **kwargs: Any) -> None

Library docstring

`Bert_score Evaluating Text Generation`_ for measuring text similarity.

BERT leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference
sentences by cosine similarity. It has been shown to correlate with human judgment on sentence-level and
system-level evaluation. Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for
evaluating different language generation tasks. This implementation follows the original implementation from
`BERT_score`_.

As input to ``forward`` and ``update`` the metric accepts the following input:

- ``preds``: Predicted sentence(s). Can be one of:

    * A single predicted sentence as a string (``str``)
    * A sequence of predicted sentences (``Sequence[str]``)

- ``target``: Target/reference sentence(s). Can be one of:

    * A single reference sentence as a string (``str``)
    * A sequence of reference sentences (``Sequence[str]``)
    * A sequence of sequences of reference sentences for multi-reference evaluation (``Sequence[Sequence[str]]``)

As output of ``forward`` and ``compute`` the metric returns the following output:

- ``score`` (:class:`~Dict`): A dictionary containing the keys ``precision``, ``recall`` and ``f1`` with
  corresponding values

Args:
    preds (Union[str, Sequence[str]]): A single predicted sentence or a sequence of predicted sentences.
    target (Union[str, Sequence[str], Sequence[Sequence[str]]]): A single target sentence, a sequence of target
        sentences, or a sequence of sequences of target sentences for multiple references per prediction.
    model_type: A name or a model path used to load ``transformers`` pretrained model.
    num_layers: A layer of representation to use.
    all_layers:
        An indication of whether the representation from all model's layers should be used.
        If ``all_layers=True``, the argument ``num_layers`` is ignored.
    model:  A user's own model. Must be of `torch.nn.Module` instance.
    user_tokenizer:
        A user's own tokenizer used with the own model. This must be an instance with the ``__call__`` method.
        This method must take an iterable of sentences (`List[str]`) and mus

Quick recipe

import torchmetrics.text as _m
score = _m.BERTScore(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).