Rougescore

rougescore

Metric ROUGEScore from torchmetrics (torchmetrics.text.ROUGEScore)

When to invoke this skill

The user has predictions + ground truth and asks to evaluate with ROUGEScore, or mentions torchmetrics.text.ROUGEScore directly, or wants the standard torchmetrics implementation.

Reference signature

from torchmetrics.text import ROUGEScore

# ROUGEScore(use_stemmer: bool = False, normalizer: Optional[Callable[[str], str]] = None, tokenizer: Optional[Callable[[str], collections.abc.Sequence[str]]] = None, accumulate: Literal['avg', 'best'] = 'best', rouge_keys: Union[str, tuple[str, ...]] = ('rouge1', 'rouge2', 'rougeL', 'rougeLsum'), **kwargs: Any) -> None

Library docstring

`Calculate Rouge Score`_, used for automatic summarization.

This implementation should imitate the behaviour of the ``rouge-score`` package `Python ROUGE Implementation`

As input to ``forward`` and ``update`` the metric accepts the following input:

- ``preds`` (:class:`~Sequence`): An iterable of predicted sentences or a single predicted sentence
- ``target`` (:class:`~Sequence`): An iterable of target sentences
  or an iterable of interables of target sentences
  or a single target sentence

As output of ``forward`` and ``compute`` the metric returns the following output:

- ``rouge`` (:class:`~Dict`): A dictionary of tensor rouge scores for each input str rouge key

Args:
    use_stemmer: Use Porter stemmer to strip word suffixes to improve matching.
    normalizer: A user's own normalizer function.
        If this is ``None``, replacing any non-alpha-numeric characters with spaces is default.
        This function must take a ``str`` and return a ``str``.
    tokenizer:
        A user's own tokenizer function. If this is ``None``, splitting by spaces is default
        This function must take a ``str`` and return ``Sequence[str]``
    accumulate:
        Useful in case of multi-reference rouge score.

        - ``avg`` takes the avg of all references with respect to predictions
        - ``best`` takes the best fmeasure score obtained between prediction and multiple corresponding references.

    rouge_keys: A list of rouge types to calculate.
        Keys that are allowed are ``rougeL``, ``rougeLsum``, and ``rouge1`` through ``rouge9``.
    kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info.

Example:
    >>> from torchmetrics.text.rouge import ROUGEScore
    >>> preds = "My name is John"
    >>> target = "Is your name John"
    >>> rouge = ROUGEScore()
    >>> from pprint import pprint
    >>> pprint(rouge(preds, target))
    {'rouge1_fmeasure': tensor(0.7500),
     'rouge1_precision': tensor(0.7500),
     'rouge1_recall': tensor(0.7500),
     'rouge2_fmeasure': tensor(0.),
     'rouge2_precision': tensor(0.),
     'rouge2_recall': tensor(0.),
     'rougeL_fmeasure': tensor(0.5000),
     'rougeL_precision': tensor(0.5000),
     

Quick recipe

import torchmetrics.text as _m
score = _m.ROUGEScore(y_true, y_pred)

Don'ts

• Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
• Always check the library version's argument order — sklearn is (y_true, y_pred) while torchmetrics is (preds, target).