Spartqa Eval

spartqa-eval

SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning — Mirzaee et al. (2021) (arXiv:2104.05832, 2021)

What this evaluates

Probes a model's ability to perform multi-step spatial reasoning over natural language stories. It evaluates understanding of spatial relations (e.g., near, far, containment) and tests robustness against surface-level vocabulary changes and question phrasing variations.

Datasets

• SPARTQA-HUMAN — total 1100; splits: train (600), test (500)

Metrics

• accuracy (primary) — range: [0, 1]
  • Micro-average accuracy across four question types (FB, FR, CO, YN). Computed as the proportion of correctly predicted answers over the total number of test instances.
• F1 score — range: [0, 1]
  • Binary F1 score computed specifically for the Yes/No (YN) question type to account for class imbalance.
• consistency & contrast accuracy — range: [0, 1]
  • Accuracy on modified question sets (consistency: same answer; contrast: flipped answer), evaluated only on questions already predicted correctly by the model.

Input / output format

Input: A contextual story paragraph followed by a multiple-choice question targeting spatial relations.

Output: Predicted answer choice (string or class label corresponding to the options).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(p == g for p, g in zip(predictions, gold_labels))
    return correct / len(gold_labels)

def compute_f1_binary(predictions, gold_labels):
    tp = sum(p == g == 1 for p, g in zip(predictions, gold_labels))
    fp = sum(p == 1 and g == 0 for p, g in zip(predictions, gold_labels))
    fn = sum(p == 0 and g == 1 for p, g in zip(predictions, gold_labels))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Excluded 'near to' and 'far from' relations from evaluation due to high annotator variance.
• High accuracy on the 'Seen' test set often reflects vocabulary overfitting rather than true spatial reasoning; the 'Unseen' set is required to verify generalization.
• Yes/No (YN) questions are highly imbalanced, making accuracy misleading; F1 is required for fair evaluation.
• Consistency and contrast metrics are only computed on questions already predicted correctly, which can overestimate model reliability.

Evidence (verbatim from paper)

Table 2 shows performance on SPARTQA-HUMAN in a low-resource setting, where 0.6k QA pairs from SPARTQA-HUMAN are used for fine-tuning these LMs and 0.5k for testing (see Table 1 for information on this split). To verify it, we compute the F1 score for YN Q-TYPE in Table 3, where we see all systems effectively achieve better scores than the majority baseline.

Citation

@misc{mirzaee2021spartqa,
  title={SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning},
  author={Mirzaee et al. (2021)},
  year={2021},
  note={arXiv:2104.05832}
}

• arXiv: 2104.05832