Occam Eval

occam-eval

Are We Done with Object-Centric Learning? — Rubinstein et al. (2025) (arXiv:2504.07092, 2025)

What this evaluates

Evaluates whether object-centric representations derived from zero-shot segmentation masks enable robust zero-shot classification under spurious background correlations, and compares them against slot-based OCL methods on unsupervised object discovery.

Datasets

• Movi-C — total 1000; splits: test (1000)
• Movi-E — total 1000; splits: test (1000)
• UrbanCars — total ?; splits: test (-1)
• ImageNet-D — total ?; splits: test (-1)
• ImageNet-9 — total ?; splits: test (-1)
• Waterbirds — total ?; splits: test (-1)
• CounterAnimals — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of samples.
• worst group accuracy (WGA) — range: [0, 1]
  • The minimum accuracy computed across predefined spurious correlation groups (e.g., background types). It measures robustness by penalizing models that rely on spurious cues.
• FG-ARI — range: [0, 1]
  • Foreground adjusted Rand index: a clustering similarity metric that evaluates how well predicted instance masks match ground truth foreground objects, ignoring background pixels.
• mBO — range: [0, 1]
  • Mean best overlap: computes the maximum intersection-over-union (IoU) between predicted and ground truth masks for each object, averaged across objects and accounting for background pixels.

Input / output format

Input: Input images containing foreground objects and spurious background cues (or synthetic scenes for object discovery). For the OCCAM pipeline, the model receives the original image and predicted object masks.

Output: Class predictions (zero-shot classification) or mask-based feature representations. For object discovery, binary/instance masks per object.

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)

def compute_wga(preds, gold, groups):
    group_accs = {}
    for p, g, grp in zip(preds, gold, groups):
        group_accs.setdefault(grp, []).append(p == g)
    return min(sum(accs)/len(accs) for accs in group_accs.values())

Common pitfalls

• Assuming slot-based OCL is required for object-centric representations, whereas zero-shot segmentation masks achieve superior performance with far fewer training samples.
• Confusing standard accuracy with Worst Group Accuracy (WGA); WGA specifically measures robustness by taking the minimum accuracy across spurious background groups, making it sensitive to background correlations.
• Overlooking that foreground mask selection remains a critical bottleneck; even with high-quality segmentation, poor foreground detection degrades downstream classification performance.

Evidence (verbatim from paper)

We measure model performance using the standard metric used in the respective benchmark: accuracy and worst group accuracy (WGA). We provide per-benchmark comparisons for reference, including results from other relevant methods, citing them alongside their names in the tables.

Citation

@misc{rubinstein2025objectcentric,
  title={Are We Done with Object-Centric Learning?},
  author={Rubinstein et al. (2025)},
  year={2025},
  note={arXiv:2504.07092}
}

• arXiv: 2504.07092