Maskeval

maskeval

MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification — Liu et al. (2022) (arXiv:2205.12394, 2022)

What this evaluates

Evaluates a reference-less, masked language model-based metric's ability to predict human judgments on text summarization and simplification quality. It probes the model's capacity to capture multiple quality dimensions such as fluency, consistency, coherence, relevance, simplicity, and meaning preservation without relying on reference texts.

Datasets

• SummEval — total 1600; splits: train (700), test (900)
• ASSET — total 162; splits: train (62), test (100)

Metrics

• pearson_correlation (primary) — range: [-1, 1]
  • Computes the Pearson correlation coefficient between the predicted scores from MaskEval and the average human-annotated scores for each quality dimension.

Input / output format

Input: Source text and candidate summary or simplification text.

Output: A scalar quality score (or dimension-specific scores) ranging from 0 to 1.

Scoring recipe

def compute_pearson(pred_scores, human_scores):
    n = len(pred_scores)
    mean_p = sum(pred_scores) / n
    mean_h = sum(human_scores) / n
    cov = sum((p - mean_p) * (h - mean_h) for p, h in zip(pred_scores, human_scores))
    std_p = (sum((p - mean_p)**2 for p in pred_scores) / n) ** 0.5
    std_h = (sum((h - mean_h)**2 for h in human_scores) / n) ** 0.5
    return cov / (std_p * std_h) if std_p * std_h > 0 else 0.0

Common pitfalls

• MaskEval is reference-less, so direct comparison with reference-based metrics like ROUGE or BLEU requires careful interpretation of the evaluation setup.
• The weighters are trained on small subsets (700 for SummEval, 62 for ASSET), making performance sensitive to the specific train/test split and potentially limiting generalization.
• Human scores are averaged across multiple annotators, so the metric's correlation reflects agreement with the mean judgment rather than individual annotator variance.

Evidence (verbatim from paper)

We evaluate MaskEval on English summarization and simplification. It contains 1,600 summary-article pairs, each pair scored by three annotators with respect to four dimensions: consistency (con), coherence (coh), fluency (flu), and relevance (rel). Table 1: English summarization results on the SummEval dataset (Pearson correlation).

Citation

@misc{liu2022maskeval,
  title={MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2205.12394}
}

• arXiv: 2205.12394