Mlissard Eval

mlissard-eval

MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks — Bueno et al. (2024) (arXiv:2410.06396, 2024)

What this evaluates

This benchmark evaluates a model's ability to perform simple sequential reasoning tasks (e.g., counting, copying, list intersection) while extrapolating to longer input sequences. It specifically probes length generalization and the impact of multilingual in-context examples on reasoning robustness across different languages.

Datasets

• MLissard — total ?; splits: test (-1); repo https://github.com/unicamp-dl/Lissard

Metrics

• accuracy (primary) — range: percent
  • Standard exact-match accuracy: the proportion of instances where the model's generated output exactly matches the gold label. Reported as a percentage or decimal in [0, 1].

Input / output format

Input: Natural language instructions and few-shot examples in various languages (English, Portuguese, Spanish, German, Russian, Ukrainian), followed by a test sequence containing a specific number of key entities/objects. Tasks include Object Counting, Last Letter Concatenation, Repeat Copy Logic, and List Intersection.

Output: The model must generate the exact expected output sequence or value corresponding to the task (e.g., a number, a concatenated string of letters, a copied list, or an intersected list).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred.strip() == gold.strip())
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Evaluating only on short sequences (Bin 1) masks the benchmark's core purpose of testing length extrapolation, as performance drops sharply in Bins 3-4.
• Task difficulty is highly uneven; 'List Intersection' and 'Repeat Copy Logic' are significantly harder than 'Object Counting', so reporting a single average without stratifying by task or bin can be misleading.
• Prompt language composition matters: using monolingual in-context examples yields lower accuracy than multilingual ones, so evaluation should control for or report prompt language variations.

Evidence (verbatim from paper)

Overall, there is a gradual decline in the performance of language models across tasks as complexity increases, as measured by the number of key entities in the input sequence. For instance, in the “Object Counting” task, when presented with inputs containing 1 to 7 objects, the model achieve approximately 100% accuracy. However, their accuracy drops below 50% when confronted with sequences with 12 to 17 objects.

Citation

@misc{bueno2024mlissard,
  title={MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks},
  author={Bueno et al. (2024)},
  year={2024},
  note={arXiv:2410.06396}
}

• arXiv: 2410.06396