Mmwp Eval

mmwp-eval

LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Reasoning — Zhang et al. (2024) (arXiv:2412.12499, 2024)

What this evaluates

Evaluates large language models' ability to perform mathematical, commonsense, and natural language inference reasoning in low-, medium-, and high-resource languages. It specifically probes cross-lingual transfer capabilities using a zero-shot chain-of-thought setting without requiring parallel multilingual instruction data.

Datasets

• MMWP — total ?; splits: test (-1)
• MGSM — total ?; splits: test (-1)
• MSVAMP — total ?; splits: test (-1)
• X-CSQA — total ?; splits: test (-1)
• XNLI — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • The proportion of correctly predicted answers out of the total number of instances, reported as a percentage. Results are aggregated by language resource level (low, medium, high) and averaged across all languages.

Input / output format

Input: Multilingual natural language problem statements (math word problems, commonsense questions, or NLI premises/hypotheses) provided in the target language.

Output: Generated text containing reasoning steps followed by a final answer.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if normalize(pred) == normalize(gold))
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Results are stratified by language resource level (LR/MR/HR) rather than reported as a single global average, which can obscure performance on specific languages.
• The zero-shot chain-of-thought setting does not provide task-specific prompts or examples, making performance highly sensitive to the model's inherent multilingual reasoning capabilities.
• Baselines like Translate-En and LangBridge use external translation tools or hidden-state replacements, which deviates from the pure zero-shot evaluation setting and complicates direct comparison.

Evidence (verbatim from paper)

We use the MMWP and the latest multilingual benchmarks, MGSM (Shi et al., [2023]) and MSVAMP (Chen et al., [2023]) to evaluate the performance of LLMs in multilingual mathematical reasoning through zero-shot chain-of-thought reasoning (Wei et al., [2022]) setting. Table 1: Results on the MMWP benchmark, where “LR.”, “MR.”, and “HR.” denote mean accuracy for low-, medium-, and high-resource languages, respectively.

Citation

@misc{zhang2024lingualift,
  title={LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Reasoning},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2412.12499}
}

• arXiv: 2412.12499