Pmmeval Eval

pmmeval-eval

P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs — Zhang et al. (2024) (arXiv:2411.09116, 2024)

What this evaluates

Evaluates multilingual capabilities of LLMs across understanding, reasoning, and generation tasks in 10 languages. It probes prompt sensitivity and cross-lingual performance consistency to reveal benchmark origin bias and language-specific scaling trends.

Datasets

• MMMLU — total ?; splits: test (-1)
• MLogiQA — total ?; splits: test (-1)
• MGSM — total ?; splits: test (-1)
• MHellaSwag — total ?; splits: test (-1)
• XNLI — total ?; splits: test (-1)
• Flores-200 — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers out of total instances. Calculated as (number of exact matches between model output and gold label) / (total number of instances) * 100.
• BLEU — range: percent
  • Standard n-gram overlap metric for machine translation evaluation, typically computed as BLEU-4 with sentence-level averaging.
• COMET — range: [0, 1]
  • Reference-based neural translation quality score computed using the wmt22-comet-da model, outputting a score typically in [0, 1].

Input / output format

Input: Instruction prompt (English, target language, or English with few-shot demonstrations) concatenated with the input instance in the target language.

Output: Direct answer string, or step-by-step Chain of Thought reasoning followed by a final answer (required for MGSM and used for small LLMs on MMMLU).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100

def compute_bleu(references, predictions):
    return bleu_score(references, predictions) * 100

def compute_comet(references, predictions):
    return comet_model.score(references, predictions)

Common pitfalls

• Prompt choice drastically changes scores (EN vs Native vs Few-shot), especially for generation tasks where English instructions cause models to output English.
• Chain-of-Thought reasoning causes high answer extraction failure rates on models with fewer than 7B parameters.
• Performance varies significantly based on whether the benchmark originates from English or Chinese training data, revealing benchmark sensitivity.

Evidence (verbatim from paper)

For the Flores-200 dataset, in addition to reporting BLEU scores, we also provide COMET scores measured by wmt22-comet-da [Rei et al.] (see Appendix, Table [5]).

Citation

@misc{zhang2024pmmeval,
  title={P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2411.09116}
}

• arXiv: 2411.09116