Lingoly Eval

lingoly-eval

LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages — Bean et al. (2024) (arXiv:2406.06196, 2024)

What this evaluates

This benchmark evaluates large language models' ability to perform multi-step linguistic reasoning and deductive puzzle solving in low-resource and extinct languages. It probes out-of-domain grammatical inference and instruction-following under conditions of minimal pre-training exposure, requiring models to extract and apply novel rules from provided context rather than relying on memorized knowledge.

Datasets

• LINGOLY — total 1133; splits: test (1133); repo https://github.com/am-bean/lingOly

Metrics

• Exact Match (primary) — range: [0, 1]
  • Binary score (1 or 0) indicating whether the model's predicted answer string exactly matches the official marking sheet answer. Non-linguistic string differences (e.g., unicode encodings) are normalized before comparison. Questions with fuzzy/synonym answers are excluded from this metric.
• Δ_NC — range: other
  • Measures the model's ability to utilize provided context by calculating the difference in exact-match accuracy between the full prompt (with context) and a no-context baseline prompt: Δ_NC = S(r_Full) - S(r_NC).

Input / output format

Input: A puzzle prompt containing a preamble, linguistic context/rules, and a question. For the no-context baseline evaluation, the context section is removed from the prompt.

Output: A single answer string corresponding to the puzzle solution. Models are instructed to format outputs as JSON.

Scoring recipe

def score_exact_match(predictions, gold_answers):
    correct = 0
    total = 0
    for pred, gold in zip(predictions, gold_answers):
        if gold is None: continue
        if normalize_unicode(pred) == normalize_unicode(gold):
            correct += 1
        total += 1
    return correct / total if total > 0 else 0.0

def compute_delta_nc(full_acc, no_context_acc):
    return full_acc - no_context_acc

Common pitfalls

• Fuzzy questions accepting synonyms or free-text are automatically excluded from the exact-match metric, so evaluators must filter them out first.
• Unicode normalization is required before comparison; raw string matching will fail on equivalent characters encoded differently.
• The no-context baseline removes the linguistic context/rules from the prompt, not just the question, to test for memorization rather than reasoning.

Evidence (verbatim from paper)

Exact Match We exclude all questions where the answer is "fuzzy" (i.e., accepts synonyms or free text response) because we cannot automate the evaluation of synonym similarity across languages. For remaining questions, we only accept the exact answer on the marking sheet. In some languages (e.g. with free word ordering), multiple answers cannot be avoided. Here, the answer key is an exhaustive list of solutions. We normalise non-linguistic differences between strings, such as unicode encodings, before evaluating matches.

Citation

@misc{bean2024lingoly,
  title={LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages},
  author={Bean et al. (2024)},
  year={2024},
  note={arXiv:2406.06196}
}

• arXiv: 2406.06196