Gscan Eval

gscan-eval

A Benchmark for Systematic Generalization in Grounded Language Understanding — Ruis et al. (2020) (arXiv:2003.05161, 2020)

What this evaluates

Evaluates systematic generalization in grounded language understanding by testing whether models can interpret natural language commands within dynamic grid-world environments. It probes compositional generalization across novel object properties, navigation directions, contextual size references, action-argument bindings, and adverbial modifiers.

Datasets

• gSCAN — total ?; splits: random (19282), novel-object-properties-red (-1), novel-object-properties-yellow (-1), novel-direction (-1), novel-contextual-references (-1), novel-action-arguments (-1), novel-adverbs-fewshot (-1), novel-adverbs-combination (-1); repo https://github.com/LauraRuis/groundedSCAN

Metrics

• exact match accuracy (primary) — range: percent
  • Calculated as the percentage of test examples where the model's predicted action sequence exactly matches the gold action sequence. Partial matches or step-wise correctness do not receive credit.

Input / output format

Input: A natural language instruction (e.g., 'walk to the small red circle') paired with a dynamic grid-world visual state containing objects of varying colors, shapes, and sizes at specific coordinates.

Output: A sequence of discrete navigation/interaction actions (e.g., walk, L_turn, R_turn, stay, push, pull) that moves the agent from its starting position to the target object and performs the required interaction.

Scoring recipe

def compute_exact_match(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Exact match requires the entire action sequence to be identical; step-wise accuracy or partial path completion is not counted as correct.
• Splits are grounded in specific world-state configurations (e.g., relative size, object weight, spatial direction), not just linguistic variations, so models cannot rely on purely syntactic generalization.
• The benchmark uses synthetic grid-world environments, so performance does not directly transfer to real-world robotics or complex visual grounding tasks without adaptation.

Evidence (verbatim from paper)

The baseline model achieves near perfect exact match accuracy ($97.69%\pm 0.22$ mean over 3 runs, reported with standard deviation) on the 19,282 test examples, where exact match means that the entire action sequence is produced correctly.

Citation

@misc{ruis2020gscan,
  title={A Benchmark for Systematic Generalization in Grounded Language Understanding},
  author={Ruis et al. (2020)},
  year={2020},
  note={arXiv:2003.05161}
}

• arXiv: 2003.05161