Wals Metalinguistic Eval

wals-metalinguistic-eval

Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages — Arčon et al. (2026) (arXiv:2602.02182, 2026)

What this evaluates

Probes large language models' ability to recall and identify structural and grammatical properties of languages across diverse linguistic domains. It measures whether models have internalized typological facts from the World Atlas of Language Structures (WALS) by answering multiple-choice questions about specific language features.

Datasets

• WALS — total 76475; splits: train (-1), val (-1), test (-1); repo https://github.com/Oranzna/metalinguistic_benchmark
• WALS-100 — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/Oranzna/metalinguistic_benchmark

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy for multiple-choice QA: 1 if the model's predicted answer exactly matches the ground-truth WALS value, 0 otherwise. Averaged across all language-feature instances.

Input / output format

Input: A multiple-choice question template instantiated for a specific language and WALS feature, accompanied by a fixed set of possible answer options derived from the feature's value categories.

Output: The model must select exactly one answer option from the provided list.

Scoring recipe

correct = 0
total = 0
for instance in dataset:
    model_pred = generate_answer(instance.prompt)
    if model_pred == instance.ground_truth:
        correct += 1
    total += 1
return correct / total

Common pitfalls

• Splitting by feature rather than language means models may memorize feature templates rather than generalizing to unseen languages.
• Annotation sparsity in WALS can confound genuine cross-linguistic performance differences with missing data.
• Surface-level memorization of terminology-heavy value labels may inflate scores if questions are not carefully rephrased.

Evidence (verbatim from paper)

accuracy in identifying grammatical features from the World Atlas of Language Structures (WALS) being strongly correlated with digital language resources rather than linguistic universals.

Citation

@misc{arcon2026metalinguistic,
  title={Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages},
  author={Arčon et al. (2026)},
  year={2026},
  note={arXiv:2602.02182}
}

• arXiv: 2602.02182