Mdia Eval

mdia-eval

MDIA: A Benchmark for Multilingual Dialogue Generation in 46 Languages — Qingyu Zhang et al. (2022) (arXiv:2208.13078, 2022)

What this evaluates

This benchmark evaluates a model's ability to generate coherent, contextually appropriate, and lexically diverse dialogue responses across 46 languages. It specifically probes cross-lingual transfer capabilities and measures the performance gap between high-resource and low-resource languages in open-domain conversation.

Datasets

• MDIA — total ?; splits: test (-1); repo https://github.com/DoctorDream/mDIA

Metrics

• sacreBLEU (primary) — range: [0, 100]
  • Standard BLEU score computed with sentence-level tokenization, casing normalization, and no smoothing, as implemented in the sacreBLEU library.
• BertScore — range: [0, 1]
  • Cosine similarity between contextual token embeddings of the generated response and reference using a multilingual BERT model, averaged across precision, recall, and F1.
• Distinct-n — range: [0, 1]
  • Ratio of unique n-grams to the total number of n-grams in the generated text, measuring lexical variety.
• Entropy — range: other
  • Shannon entropy calculated over the probability distribution of n-grams in the generated text.
• SSA — range: [0, 1]
  • Sensibleness and Specificity Average, computed as the mean of human-rated scores for whether a response makes sense in context (sensibleness) and is specific to the context (specificity).

Input / output format

Input: A sequence of dialogue turns (context) in a specific language.

Output: A single response turn in the same language as the context.

Scoring recipe

def score(predictions, gold):
    bleu = sacrebleu.corpus_bleu(predictions, [gold])
    bert = bertscore.score(predictions, gold, lang='multilingual')
    distinct_n = [len(set(ngrams(p, n)))/len(list(ngrams(p, n))) for p in predictions]
    entropy_n = [shannon_entropy(ngrams(p, n)) for p in predictions]
    ssa = mean([human_rate_sens(p, c) for p, c in zip(predictions, contexts)])
    return {'sacreBLEU': bleu, 'BertScore': bert, 'Distinct-n': distinct_n, 'Entropy-n': entropy_n, 'SSA': ssa}

Common pitfalls

• Automatic metrics like sacreBLEU and BertScore are not directly comparable across different languages due to variations in tokenizer design and underlying model architectures.
• Human evaluation scores (SSA) do not correlate well with automatic metric scores, meaning high BLEU/BertScore does not guarantee high human-perceived quality.
• The 'high-resource' vs 'low-resource' classification is strictly based on Reddit conversation counts (≥12k) and MarianMT translation availability, not general linguistic resource availability.

Evidence (verbatim from paper)

We evaluate the quality of generated responses with sacreBLEU (Papineni et al., 2002) and BertScore (Zhang et al., 2020a). Both of them are realized on multilingual tokenizers and fixed parameters, ensuring the reproducibility and the comparability of our benchmark. We also use Entropy (Zhang et al., 2018) and Dist-n (Li et al., 2016; Su et al., 2018) to evaluate lexical diversity.

Citation

@misc{zhang2022mdia,
  title={MDIA: A Benchmark for Multilingual Dialogue Generation in 46 Languages},
  author={Qingyu Zhang et al. (2022)},
  year={2022},
  note={arXiv:2208.13078}
}

• arXiv: 2208.13078