Unisumeval Eval

unisumeval-eval

UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs — Lee et al. (2024) (arXiv:2409.19898, 2024)

What this evaluates

Evaluates text summarization models across multiple dimensions including faithfulness, completeness, conciseness, domain stability, and abstractiveness. It tests how well summarizers handle diverse input contexts (domains, dialogue vs. non-dialogue, short vs. long texts) and the impact of PII redaction on hallucination.

Datasets

• UniSumEval — total ?; splits: test (-1); repo https://github.com/DISL-Lab/UniSumEval-v1.0

Metrics

• faithfulness (primary) — range: percent
  • Percentage score computed from fine-grained human annotations measuring whether the summary contains information not present in or contradicting the source text.
• completeness — range: percent
  • Percentage score computed from fine-grained human annotations measuring the extent to which the summary covers the key facts and main ideas of the source text.
• conciseness — range: percent
  • Percentage score computed from fine-grained human annotations measuring whether the summary is concise and avoids redundant or unnecessary information.
• domain stability — range: percent
  • Composite score calculated as the average of faithfulness, completeness, and conciseness scores; domain inconsistency is measured by the gap between the highest and lowest composite scores across the nine domains.
• abstractiveness — range: other
  • Average count of novel 1-gram, 3-gram, and 5-gram phrases in the summary compared to the source text, following Song et al. (2023).

Input / output format

Input: Source text (document or dialogue) varying by domain (9 types), type (dialogue/non-dialogue), length (short/long, up to 10,462 words), and PII status (redacted/unredacted).

Output: Generated summary text.

Scoring recipe

def compute_scores(predictions, gold_annotations):
    scores = {}
    for dim in ['faithfulness', 'completeness', 'conciseness']:
        correct = sum(1 for a in gold_annotations if a[dim] == 'correct')
        scores[dim] = (correct / len(gold_annotations)) * 100
    composites = [avg(scores[d]) for d in domains]
    scores['domain stability'] = max(composites) - min(composites)
    scores['abstractiveness'] = mean(novel_ngrams(predictions, gold_annotations['source'], n=[1,3,5]))
    return scores

Common pitfalls

• Conciseness evaluation yields significantly lower scores and is harder to assess reliably than faithfulness or completeness.
• PII redaction in input texts exacerbates hallucination, particularly causing non-LLM summarizers to invent or misrepresent masked entities.
• Non-LLM automated evaluators (e.g., QA-based, NLI-based) show poor correlation with human faithfulness scores and fail to detect hallucinations in LLM-generated summaries.

Evidence (verbatim from paper)

We report percentage scores (in Section [3.3]) of faithfulness, completeness, and conciseness, computed by using fine-grained human annotations. For domain stability, we calculate the average of the three percentage scores to obtain a composite score, and then measure domain inconsistency by computing the gap between the highest and lowest composite ones. For abstractiveness, we use the average of novel 1/3/5-grams following Song et al. ([2023]).

Citation

@misc{lee2024unisumeval,
  title={UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs},
  author={Lee et al. (2024)},
  year={2024},
  note={arXiv:2409.19898}
}

• arXiv: 2409.19898