Norsumm Eval

norsumm-eval

Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles — Touileb et al. (2025) (arXiv:2501.07718, 2025)

What this evaluates

This benchmark evaluates the abstractive summarization capabilities of LLMs on Norwegian news articles. It specifically probes models' ability to generate concise, accurate, and linguistically appropriate summaries in both Bokmål and Nynorsk written variants, while preserving key information and cultural nuance.

Datasets

• NorSumm — total ?; splits: test (-1); repo https://github.com/SamiaTouileb/NorSumm

Metrics

• BERTScore (primary) — range: [0, 1]
  • Computes the cosine similarity between contextual embeddings of generated and reference tokens, then averages across tokens. The protocol maximizes this score across multiple prompts before averaging.
• ROUGE-L — range: [0, 1]
  • Measures the longest common subsequence overlap between the generated summary and the reference summary, capturing sentence-level fluency and content coverage.

Input / output format

Input: A prompt template in either Bokmål or Nynorsk followed by the full source news article text. Six distinct prompt styles are used per language variety, varying instruction wording, verbosity, and article placement.

Output: Free-form text summary generated via greedy search decoding. Prompts specify a maximum length of 700 characters including spaces.

Scoring recipe

# For each article and language variety:
# 1. Generate summary for each of the 6 prompts using greedy decoding.
# 2. For each prompt, compute ROUGE-L and BERTScore against 3 human references.
# 3. Keep the maximum ROUGE-L and BERTScore across the 3 references per prompt.
# 4. Select the prompt that yields the highest BERTScore.
# 5. Average the selected BERTScore and corresponding ROUGE-L across all articles.

Common pitfalls

• Prompt sensitivity: Simply averaging scores across all prompts or using a single prompt will misrepresent model capability due to varying instruction styles.
• Reference selection: The protocol requires taking the maximum score against three human references per instance, not averaging or picking the first reference.
• Decoding method: Summaries must be generated via greedy search, not sampling or beam search, to match the reported baseline.

Evidence (verbatim from paper)

We measure the performance using standard summarisation evaluation metrics: ROUGE-L and BERTScore. Our result aggregation procedure accounts for prompt sensitivity and includes two steps: (i) for each prompt, we compute the maximum performance scores between the LLM’s output and each of three human-written references (our human-authored summaries); (ii) we then maximize the BERTScore across all prompts and average the resulting ROUGE-L and BERTScore values over all BM/NN examples.

Citation

@misc{touileb2025norsumm,
  title={Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles},
  author={Touileb et al. (2025)},
  year={2025},
  note={arXiv:2501.07718}
}

• arXiv: 2501.07718