Multinrc Eval

multinrc-eval

MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs — Fabbri et al. (2025) (arXiv:2507.17476, 2025)

What this evaluates

This benchmark evaluates LLMs' ability to perform multi-step reasoning in native non-English languages (French, Spanish, Chinese) across linguistic, wordplay, cultural/tradition, and culturally-grounded math categories. It specifically probes whether models rely on translation bias or possess deep cultural and linguistic contextual knowledge required for accurate problem-solving.

Datasets

• MultiNRC — total 1055; splits: test (1055)

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy computed by comparing the model's predicted final answer against the annotator-provided Ground-Truth Final Answer (GTFA). A prediction is counted as correct only if it matches the GTFA in meaning and format (single word or short phrase/sentence).

Input / output format

Input: A reasoning prompt in French, Spanish, or Chinese belonging to one of four categories: language-specific linguistic reasoning, wordplay/riddles, cultural/tradition reasoning, or math reasoning with cultural relevance. Some cultural/math prompts also include an English-translated equivalent.

Output: A single word or short phrase/sentence representing the final answer to the reasoning prompt.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize(pred.strip()) == normalize(gold.strip()):
            correct += 1
    return correct / len(predictions) if predictions else 0.0

Common pitfalls

• Assuming English-translated prompts are equivalent to native ones; wordplay and linguistic categories explicitly cannot be reliably translated without losing the core reasoning mechanism.
• Treating cultural/math questions as pure arithmetic or logic problems; they require applying culturally specific knowledge (e.g., local holidays, currency systems, real estate conventions) to reach the correct answer.
• Ignoring the multi-step reasoning requirement; questions are filtered to ensure they demand more than mere feature identification or single-step calculation.

Evidence (verbatim from paper)

For each prompt, the annotator also writes a ground-truth final answer (GTFA), consisting of either a single word or a short phrase/sentence. We only keep the reasoning questions that 3 or more of the 5 models fail to correctly answer. After the raw data is collected, we have two native-speaker reviewer layers to assess the quality of the data from different aspects. The first native-speaker reviewer layer is to assess the quality of the prompt and GTFA pairs. At this layer we would assess both the GTFA’s accuracy and the prompt’s reasonableness and alignment with category definition.

Citation

@misc{fabbri2025multinrc,
  title={MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs},
  author={Fabbri et al. (2025)},
  year={2025},
  note={arXiv:2507.17476}
}

• arXiv: 2507.17476