Eval4nlp 2023 Shared Task Eval

eval4nlp-2023-shared-task-eval

The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics — Leiter et al. (2023) (arXiv:2310.19792, 2023)

What this evaluates

Evaluates reference-free LLM prompting strategies as metrics for machine translation and summarization. It measures how well predicted quality scores correlate with human judgments (MQM for MT, human annotations for summarization).

Datasets

• Eval4NLP 2023 Shared Task (MT & Summarization) — total ?; splits: test (-1); repo https://github.com/eval4nlp/SharedTask2023

Metrics

• Kendall correlation (primary) — range: [-1, 1]
  • Measures the rank correlation between predicted metric scores and gold human scores (MQM for MT, human annotations for summarization). Computed per language pair or summarization subcategory.

Input / output format

Input: Source text and generated hypothesis/summary. For MT, the language pair (e.g., en-de) is also specified.

Output: A continuous quality score (and optionally a textual explanation).

Scoring recipe

def evaluate(predictions, gold):
    # predictions: list of float scores from LLM metric
    # gold: list of float human/MQM scores
    kendall = kendalltau(predictions, gold).correlation
    pearson = pearsonr(predictions, gold).statistic
    spearman = spearmanr(predictions, gold).correlation
    return {"kendall": kendall, "pearson": pearson, "spearman": spearman}

Common pitfalls

• Participants may over-optimize on the dev/test leaderboard by submitting many iterations, violating the intended generalization evaluation.
• Probability-based scoring requires a forward pass per token, which is computationally heavy compared to direct score prediction.
• Explanations generated by LLMs are often vague and may not faithfully reflect the reasoning behind the predicted score.

Evidence (verbatim from paper)

Each column shows the correlation of metric scores to MQM scores for English-X language pairs. Results that are bolded are significantly better than non-bolded results, with $p \leq 0.05$, as measured by a permute-both significance test (Deutsch et al., 2021). kd stands for Kendall, ps stands for Pearson and sp stands for Spearman.

Citation

@misc{leiter2023eval4nlp,
  title={The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics},
  author={Leiter et al. (2023)},
  year={2023},
  note={arXiv:2310.19792}
}

• arXiv: 2310.19792