Global Piqa Eval

global-piqa-eval

Global PIQA: Evaluating Physical Commonsense Reasoning Across 100+ Languages and Cultures — Chang et al. (2025) (arXiv:2510.24081, 2025)

What this evaluates

This benchmark probes physical commonsense reasoning by testing whether models can distinguish correct from incorrect solutions to everyday physical tasks. It specifically evaluates cultural and linguistic grounding by using items constructed natively in 116 language varieties, avoiding translation artifacts that often skew multilingual evaluations.

Datasets

• Global PIQA — total ?; splits: non-parallel (-1); repo https://github.com/mrlbenchmarks/global-piqa

Metrics

• accuracy (primary) — range: percent
  • Mark correct if the model's predicted solution matches the ground truth via exact string matching (prompted format) or if the correct candidate has a higher normalized log-probability than the incorrect candidate (completion format). Chance accuracy is 50%.

Input / output format

Input: A prompt containing a physical commonsense scenario or question in a specific language, followed by two candidate solutions (one correct, one incorrect).

Output: For prompted evaluation: a text response sampled up to 2048 tokens. For completion evaluation: log-probabilities for each candidate solution.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_sol in zip(predictions, gold):
        if pred.strip() == gold_sol.strip():
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Using the prompted format for base/pretrained models obscures true capabilities because instruction-following imposes auxiliary task demands that lower scores.
• Reporting only aggregate accuracy masks severe performance disparities across regions, with some low-resource languages dropping up to 37% below high-resource baselines.

Evidence (verbatim from paper)

For models that are tuned to follow instructions (e.g. the vast majority of proprietary models, and instruction-tuned and RL-tuned open models), we prompt the LLM with the prompt template in Figure[2]. We sample up to 2048 tokens, and score the responses using string matching. ... For all results, we report accuracy, where chance accuracy is 50%.

Citation

@misc{chang2025globalpiqa,
  title={Global PIQA: Evaluating Physical Commonsense Reasoning Across 100+ Languages and Cultures},
  author={Chang et al. (2025)},
  year={2025},
  note={arXiv:2510.24081}
}

• arXiv: 2510.24081