Unieval Eval

unieval-eval

Towards a Unified Multi-Dimensional Evaluator for Text Generation — Zhong et al. (2022) (arXiv:2210.07197, 2022)

What this evaluates

Evaluates natural language generation models across multiple quality dimensions (e.g., coherence, fluency, consistency, relevance) by reframing assessment as a Boolean QA task. Measures how well automated scores align with human judgments using correlation metrics.

Datasets

• SummEval — total ?; splits: test (-1)
• Topical-Chat — total ?; splits: test (-1)
• SFRES — total ?; splits: test (-1)
• SFHOT — total ?; splits: test (-1)
• QAGS — total ?; splits: test (-1)

Metrics

• Spearman correlation (primary) — range: [-1, 1]
  • Ranks the predicted scores and human scores, then computes the Pearson correlation between the rank vectors. Measures monotonic relationship.
• Kendall-Tau — range: [-1, 1]
  • Computes the difference between concordant and discordant pairs of predictions and human scores, normalized by total pairs. Measures ordinal association.
• Pearson correlation — range: [-1, 1]
  • Computes the covariance of predicted and human scores divided by the product of their standard deviations. Measures linear relationship.

Input / output format

Input: A targeted Boolean question for a specific evaluation dimension (e.g., 'Is this a coherent summary?'), the source/context text, and the generated text.

Output: Boolean answer (Yes/No) or probability of 'Yes'.

Scoring recipe

def compute_score(predictions, human_scores, dimension):
    if dimension in ['fluency', 'consistency']:
        score = mean(sentence_level_scores)
    elif dimension == 'engagingness':
        score = sum(sentence_level_scores)
    else:
        score = compute_equation_1(prediction, human_label)
    return spearmanr(predictions, human_scores)

Common pitfalls

• Confusing summary-level aggregation (for SummEval) with turn-level aggregation (for Topical-Chat), which changes the correlation baseline.
• Using similarity-based metrics like ROUGE or BERTScore for dimensions they were not designed for (e.g., coherence), leading to artificially low correlations.
• Ignoring the continual learning schedule (coherence -> fluency -> consistency -> relevance), which significantly impacts zero-shot transfer performance.

Evidence (verbatim from paper)

Following Liu et al. (2021a), we use summary-level Spearman and Kendall-Tau correlation to assess the performance of different evaluators for summarization.

Citation

@misc{zhong2022unieval,
  title={Towards a Unified Multi-Dimensional Evaluator for Text Generation},
  author={Zhong et al. (2022)},
  year={2022},
  note={arXiv:2210.07197}
}

• arXiv: 2210.07197