Dolphin Arabic Nlg Eval

dolphin-arabic-nlg-eval

Dolphin: A Challenging and Diverse Benchmark for Arabic NLG — Nagoudi et al. (2023) (arXiv:2305.14989, 2023)

What this evaluates

Evaluates the natural language generation capabilities of models across 13 diverse Arabic tasks, including machine translation, summarization, question generation, and dialectal normalization. It probes how well models handle linguistic variability across Classical Arabic, Modern Standard Arabic, dialects, and Arabizi, as well as cross-lingual and code-switched scenarios.

Datasets

• Dolphin — total ?; splits: test (50)

Metrics

• BLEU (primary) — range: [0, 1]
  • Computes the geometric mean of modified n-gram precisions (typically up to 4-grams) between the generated text and reference(s), penalizing brevity with a brevity penalty.
• ROUGE — range: [0, 1]
  • Measures recall-based n-gram overlap (typically ROUGE-1 and ROUGE-L) between the generated text and reference(s), widely used for summarization and generation tasks.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for n-gram or token overlap between prediction and reference.
• MaxMatch (M2) — range: [0, 1]
  • Evaluates grammatical error correction by measuring the overlap of edit operations (insertions, deletions, substitutions) between the system output and reference corrections.
• Character Error Rate (CER) — range: [0, 1]
  • Calculates the minimum number of character-level insertions, deletions, and substitutions required to transform the predicted string into the reference string, normalized by reference length.

Input / output format

Input: Varies by task cluster: typically a source text (sentence, paragraph, table, or code-switched text) or a passage with an answer. Models receive the input instance and must generate a target text sequence.

Output: A generated text sequence in Arabic (or target language) corresponding to the input. Format depends on the specific task (e.g., translated sentence, summary, question, paraphrase, diacritized text).

Scoring recipe

def compute_metrics(predictions, references):
    # predictions and references are lists of strings
    bleu = compute_bleu(references, predictions)
    rouge = compute_rouge(references, predictions)
    f1 = compute_f1(references, predictions)
    # For GEC tasks:
    m2 = compute_maxmatch(predictions, references)
    # For diacritization:
    cer = compute_cer(predictions, references)
    return {'BLEU': bleu, 'ROUGE': rouge, 'F1': f1, 'M2': m2, 'CER': cer}

Common pitfalls

• Multiple valid references exist for open-ended tasks (e.g., question generation, title generation), making strict exact-match or single-reference metrics misleading.
• High linguistic variability across MSA, dialects, and Arabizi can cause tokenization mismatches, artificially deflating n-gram metrics like BLEU/ROUGE.
• Several tasks (e.g., code-switching, multi-choice QA) are evaluated under zero-shot conditions without task-specific training data, requiring careful prompt formatting.

Evidence (verbatim from paper)

Standard evaluation metrics. Most generation tasks can be evaluated using traditional automated metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004). Both of these metrics evaluate the n-gram overlap between a reference text and the generated text. Nevertheless, in many tasks (e.g., question generation, open domain generation, title generation) there are multiple valid ways to produce a given text. In our benchmark, in addition to F1, BLEU, and ROUGE, we use several other evaluation metrics such MaxMatch (M2) (Dahlmeier and Ng, 2012) for grammatical error correction, and Character Error Rate (CER) (Morris et al., 2004) for diacritization.

Citation

@misc{nagoudi2023dolphin,
  title={Dolphin: A Challenging and Diverse Benchmark for Arabic NLG},
  author={Nagoudi et al. (2023)},
  year={2023},
  note={arXiv:2305.14989}
}

• arXiv: 2305.14989