Feedsum Eval

feedsum-eval

Learning to Summarize from LLM-generated Feedback — Song et al. (2024) (arXiv:2410.13116, 2024)

What this evaluates

This benchmark evaluates how well LLM-generated feedback aligns with human preferences for text summarization, and tests whether preference learning (DPO) using multi-dimensional feedback improves summary quality over supervised fine-tuning.

Datasets

• FeedSum — total 125000; splits: train (125000)

Metrics

• Spearman correlation (primary) — range: [-1, 1]
  • Computes the rank correlation coefficient between the LLM-generated feedback scores (composite or single-dimension) and human-assigned composite scores. Values closer to 1 indicate stronger alignment with human judgment.

Input / output format

Input: Document text paired with a candidate summary. For preference learning, inputs are formatted as (document, chosen_summary) vs (document, rejected_summary) pairs.

Output: For feedback generation: JSON-like scores (e.g., {Faithfulness: 5, Completeness: 3, Conciseness: 3} or percentage scores). For summarization models: a generated summary text.

Scoring recipe

def compute_spearman(llm_scores, human_scores):
    rank_llm = rankdata(llm_scores)
    rank_human = rankdata(human_scores)
    n = len(rank_llm)
    d_sq = sum((r1 - r2)**2 for r1, r2 in zip(rank_llm, rank_human))
    return 1 - (6 * d_sq) / (n * (n**2 - 1))

Common pitfalls

• The composite score used for DPO pair selection is the average across dimensions, not a sum or max.
• The 125K dataset size is a filtered subset of 182K generated pairs; documents exceeding 8K tokens or yielding malformed LLM feedback are excluded.
• Feedback configurations C1–C4 differ in LLM capacity, dimensionality, and granularity, which must be matched to the correct training setup.

Evidence (verbatim from paper)

assessed based on their Spearman correlation with human composite scores in UniSumEval... A summary is selected as the "chosen" one if it scores ≥ 4 on the Likert scale or ≥ 80% in percentage scores... Conversely, a summary is considered "rejected" if its score is at least 1 point lower on the Likert scale or 20 percentage points lower than the chosen one.

Citation

@misc{song2024learning,
  title={Learning to Summarize from LLM-generated Feedback},
  author={Song et al. (2024)},
  year={2024},
  note={arXiv:2410.13116}
}

• arXiv: 2410.13116