Compositional Arc Eval

compositional-arc-eval

Compositional-ARC: Assessing Systematic Generalization in Abstract Spatial Reasoning — Mondorf et al. (2025) (arXiv:2504.01445, 2025)

What this evaluates

This benchmark evaluates systematic generalization in abstract spatial reasoning by testing whether models can infer and compose geometric transformations (e.g., translation, rotation, reflection) from limited few-shot examples. It specifically probes out-of-distribution compositionality by training on known transformation primitives and level-1 compositions, then testing on novel level-2 compositions.

Datasets

• Compositional-ARC — total 100000; splits: train (82908), val (8546), test (8546); repo https://github.com/mainlp/C-ARC

Metrics

• exact match accuracy (primary) — range: percent
  • 1 if the predicted output grid exactly matches the target grid cell-by-cell, else 0. Averaged across episodes.
• color accuracy — range: percent
  • Percentage of predicted objects that match the target colors, ignoring shape and spatial location.
• shape accuracy — range: percent
  • Percentage of predicted objects that match the target shapes, ignoring color and spatial location.

Input / output format

Input: Few-shot study examples (3 for 3-Shot setup, 12 for Systematicity setup) representing input-output grid pairs, followed by a query input grid. Grids are provided as 2D arrays (text) or images.

Output: A single output grid matching the dimensions and format of the query input, representing the predicted result of applying the inferred transformation composition.

Scoring recipe

def compute_metrics(predictions, targets):
    exact = sum(1 for p, t in zip(predictions, targets) if p == t) / len(targets) * 100
    color_acc = 0.0
    shape_acc = 0.0
    for p, t in zip(predictions, targets):
        p_objs = extract_objects(p)
        t_objs = extract_objects(t)
        if len(p_objs) == len(t_objs):
            color_acc += sum(p.c == t.c for p, t in zip(p_objs, t_objs)) / len(t_objs)
            shape_acc += sum(p.s == t.s for p, t in zip(p_objs, t_objs)) / len(t_objs)
    color_acc = (color_acc / len(targets)) * 100
    shape_acc = (shape_acc / len(targets)) * 100
    return exact, color_acc, shape_acc

Common pitfalls

• The test set contains novel level-2 transformation compositions (e.g., translation+rotation+extension) that are strictly out-of-distribution relative to training compositions, making standard in-distribution accuracy misleading.
• Providing multimodal inputs (images of study examples and query) alongside text prompts can significantly degrade performance for some LLMs compared to text-only 2D array representations.
• Domain-specific models heavily rely on test-time training (TTT) and depth-first search candidate generation to achieve high scores, which are not standard inference procedures for general-purpose LLMs.

Evidence (verbatim from paper)

To evaluate the quality of the generated output grids, we use three different metrics: i) exact match accuracy, ii) color accuracy, and iii) shape accuracy. Exact match accuracy requires that a prediction is accurate only if every cell matches the target grid. Color accuracy checks whether predicted objects match target colors, ignoring shape and location. Shape accuracy checks whether predicted objects match target shapes, ignoring color and location.

Citation

@misc{mondorf2025compositionalarc,
  title={Compositional-ARC: Assessing Systematic Generalization in Abstract Spatial Reasoning},
  author={Mondorf et al. (2025)},
  year={2025},
  note={arXiv:2504.01445}
}

• arXiv: 2504.01445