Nubia Eval

nubia-eval

NUBIA: NeUral Based Interchangeability Assessor for Text Generation — Hassan Kane et al. (arXiv:2004.14667, 2020)

What this evaluates

Evaluates a learned neural metric's ability to correlate with human judgments of text generation quality. It probes semantic similarity, logical inference, and sentence likelihood capabilities across machine translation and image captioning domains.

Datasets

• WMT (Machine Translation) — total ?; splits: test_2017 (3920), test_2018 (207576), test_2019 (281009)
• Flickr 8K — total 5822; splits: test (5822)

Metrics

• Pearson correlation (primary) — range: [-1, 1]
  • Measures the linear correlation between predicted quality scores and averaged human assessment scores (0-100 scale).
• Kendall's Tau — range: [-1, 1]
  • Measures rank correlation between predicted scores and human rankings. Used for relative ranking and image captioning tasks.

Input / output format

Input: Source sentence, candidate translation/caption, and reference translation/caption.

Output: A continuous quality score (0-100 scale) predicted by the model, which is then correlated with human scores.

Scoring recipe

def compute_metric(predictions, human_scores, task_type):
    if task_type == 'direct_assessment':
        return pearsonr(predictions, human_scores)
    elif task_type in ['relative_ranking', 'image_captioning']:
        # Filter pairs where human score gap > 25 points
        valid = [(p, h) for p, h in zip(predictions, human_scores) if abs(h[0] - h[1]) > 25]
        return kendalltau([p for p, h in valid], [h for p, h in valid])

Common pitfalls

• The model is strictly trained on English target sentences due to LM constraints, so cross-lingual evaluation is not supported.
• Relative ranking evaluation explicitly discards sentence pairs where the human score difference is ≤ 25 points.
• The image captioning aggregator is not fine-tuned on captioning data, relying solely on weights trained on WMT MT data.

Evidence (verbatim from paper)

The performance of metrics is assessed using pearson correlation with human judgement. For this task, we used the 2017 dataset because, unlike the WMT 2018 and WMT 2019 dataset, each sentence has been scored by at least 15 human evaluators Ma et al. (2018). ... In that setting, metrics are scored on their ability to preserve the human ranking using the Kendall's Tau correlation coefficient.

Citation

@misc{kane2020nubia,
  title={NUBIA: NeUral Based Interchangeability Assessor for Text Generation},
  author={Hassan Kane et al.},
  year={2020},
  note={arXiv:2004.14667}
}

• arXiv: 2004.14667