Piqa Eval

piqa-eval

PIQA: Reasoning about Physical Commonsense in Natural Language — Bisk et al. (2019) (arXiv:1911.11641, 2019)

What this evaluates

Evaluates a model's ability to reason about physical commonsense and intuitive physics by selecting the correct solution for a given goal from two options. It probes understanding of object affordances, material properties, and non-prototypical uses of everyday items.

Datasets

• PIQA — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly predicted options out of the total number of instances. Computed as a binary classification accuracy over two candidate solutions.

Input / output format

Input: A goal description followed by two candidate solutions (Option A and Option B), processed with a [CLS] token for each choice.

Output: A binary classification prediction indicating whether Option A or Option B is the correct solution.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Some model 'mistakes' are actually correct but require external web search to verify, making strict accuracy evaluation potentially misleading.
• Finetuning can be highly unstable with certain hyperparameters, sometimes dropping validation performance to chance levels.
• Human performance was only evaluated on the development set, not the test set, so direct human-to-model comparison on test data is not available.

Evidence (verbatim from paper)

Table 1: Results of state-of-the-art natural language understanding models on PIQA, compared with human performance. The results show a significant gap between model and human performance, of roughly 20 absolute points. Accuracy (%) Validation Test

Citation

@misc{bisk2019piqa,
  title={PIQA: Reasoning about Physical Commonsense in Natural Language},
  author={Bisk et al. (2019)},
  year={2019},
  note={arXiv:1911.11641}
}

• arXiv: 1911.11641