Noticia Eval

noticia-eval

NoticIA: A Clickbait Article Summarization Dataset in Spanish — García-Ferrero et al. (2024) (arXiv:2404.07611, 2024)

What this evaluates

This benchmark evaluates large language models' ability to interpret misleading clickbait headlines and extract the core information buried in Spanish news articles. It probes the models' capacity for ultra-concise abstractive summarization in a multilingual setting, specifically testing whether they can ignore irrelevant article content and produce brief, accurate summaries.

Datasets

• NoticIA — total 850; splits: test (850); repo https://github.com/ikergarcia1996/NoticIA

Metrics

• ROUGE-1 (primary) — range: [0, 1]
  • Recall-oriented metric measuring the overlap of unigrams (whole words) between reference and generated summaries. Computed by lowercasing both texts and removing punctuation before calculating the ratio of overlapping words to reference words.
• average summary length — range: other
  • The mean number of words in the generated summaries across the dataset. Lower values are preferred to ensure conciseness.

Input / output format

Input: An instruction prompt defining the task and annotation guidelines, followed by a clickbait headline and the full article body (averaging 550 words).

Output: A concise Spanish summary (ultrasummary) of the article, generated using greedy decoding.

Scoring recipe

def compute_rouge1(reference, prediction):
    ref_words = set(reference.lower().replace('.', '').replace(',', '').split())
    pred_words = set(prediction.lower().replace('.', '').replace(',', '').split())
    if not ref_words:
        return 0.0
    overlap = len(ref_words & pred_words)
    return overlap / len(ref_words)

# For each instance:
rouge_scores = [compute_rouge1(gold, pred) for gold, pred in zip(gold_summaries, predictions)]
avg_rouge = sum(rouge_scores) / len(rouge_scores)
avg_length = sum(len(pred.split()) for pred in predictions) / len(predictions)

Common pitfalls

• Standard summarization metrics often reward verbosity; here, longer summaries may artificially inflate ROUGE scores while violating the task's strict conciseness requirement.
• The evaluation explicitly penalizes including article details that do not directly answer the clickbait headline, a nuance not captured by standard ROUGE which only measures word overlap.
• All models are evaluated using greedy search to ensure consistency, which may disadvantage models that perform better with sampling or temperature-based decoding.

Evidence (verbatim from paper)

As standard in summarization tasks, we use the ROUGE score metric (Lin, 2004) to automatically evaluate the summaries produced by the models. ROUGE is a recall-oriented summarization metric that assesses the quality of summarization systems by determining how much of the basic units in the reference summaries appear in the machine-generated summaries. Our primary metric is ROUGE-1, which considers whole words as the basic units. To compute the ROUGE score, we lowercase both summaries and remove punctuation.

Citation

@misc{garciaferrero2024noticia,
  title={NoticIA: A Clickbait Article Summarization Dataset in Spanish},
  author={García-Ferrero et al. (2024)},
  year={2024},
  note={arXiv:2404.07611}
}

• arXiv: 2404.07611