Mebench Eval

mebench-eval

MEBench: A Novel Benchmark for Understanding Mutual Exclusivity Bias in Vision-Language Models — Anh Thai et al. (arXiv:2505.20122, 2025)

What this evaluates

Evaluates vision-language models' ability to ground objects and exhibit mutual exclusivity bias when mapping novel pseudo-labels to unknown items in cluttered scenes. It also measures spatial reasoning capabilities and the model's ability to resolve ambiguity among multiple novel objects.

Datasets

• MEBench — total ?; splits: eval (-1); repo https://github.com/ngailapdi/MEBench

Metrics

• AP@t — range: [0, 1]
  • Average Precision at a specified Intersection over Union (IoU) threshold, standard for object detection tasks.
• ME score (primary) — range: [-1, 1]
  • Normalized score measuring mutual exclusivity bias: (p(novel->novel|known->known) - p(novel->known|known->known)) / (p(novel->novel|known->known) + p(novel->known|known->known)). Range is [-1, 1].
• Spatial Reasoning — range: percent
  • Percentage increase in correct novel label assignment when spatial context is provided versus absent: (p_w - p_w/o) / p_w/o.
• Ambiguity — range: [0, 1]
  • Error rate of misassigning a novel label to the wrong novel object given correct known-object classification: p(novel->wrong_novel) / (p(novel->novel) + p(novel->wrong_novel)).

Input / output format

Input: RGB image of a procedurally generated 3D scene and a text prompt specifying an object label (either a known real-world label or a novel pseudo-label).

Output: Bounding box or segmentation mask for the target object, plus a categorical label assignment decision (novel object, known object, background, or refusal to predict).

Scoring recipe

# Condition on correct known-object classification (x_k->k)
known_correct_count = count(predictions where known object is correctly classified)

# Count outcomes for novel labels given known objects are correct
p_n_to_n = count(novel label correctly assigned to novel object) / known_correct_count
p_n_to_k = count(novel label incorrectly assigned to known object) / known_correct_count

# Compute ME score
if (p_n_to_n + p_n_to_k) > 0:
    me_score = (p_n_to_n - p_n_to_k) / (p_n_to_n + p_n_to_k)
else:
    me_score = 0.0
return me_score

Common pitfalls

• Failing to condition metrics on correct known-object classification (x_k->k), which inflates or deflates the true ME bias measurement.
• Ignoring model refusal to predict or background misclassifications, which are explicitly tracked as separate outcomes that lower the effective ME score.
• Computing the Spatial Reasoning metric without explicitly comparing performance with vs. without spatial context descriptions.

Evidence (verbatim from paper)

To evaluate ME bias, we focus on conditional metrics given correct known-object classification: $p(x_{n\rightarrow n}|x_{k\rightarrow k})$: correct novel label assignment; and $p(x_{n\rightarrow k}|x_{k\rightarrow k})$: incorrect novel label assignment to a known object. An effective model should maximize $p(x_{n\rightarrow n}|x_{k\rightarrow k})$ while minimizing $p(x_{n\rightarrow k}|x_{k\rightarrow k})$. We capture this trade-off using a normalized ME score: $-1\leq\text{ME}=\frac{p(x_{n\rightarrow n}|x_{k\rightarrow k})-p(x_{n\rightarrow k}|x_{k\rightarrow k})}{p(x_{n\rightarrow n}|x_{k\rightarrow k})+p(x_{n\rightarrow k}|x_{k\rightarrow k})}\leq 1$

Citation

@misc{thai2025mebench,
  title={MEBench: A Novel Benchmark for Understanding Mutual Exclusivity Bias in Vision-Language Models},
  author={Anh Thai et al.},
  year={2025},
  note={arXiv:2505.20122}
}

• arXiv: 2505.20122