Mc Bench Eval

mc-bench-eval

MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs — Xu et al. (2024) (arXiv:2410.12332, 2024)

What this evaluates

This benchmark evaluates multi-context visual grounding, requiring models to localize target objects across multiple images using open-ended, context-rich text prompts. It probes cross-image reasoning, fine-grained instance localization, and the ability to correctly group and reject irrelevant instances.

Datasets

• MC-Bench — total 2000; splits: test (2000)

Metrics

• Accuracy (Acc) — range: percent
  • Binary metric checking whether the model correctly identifies which images contain the target objects described in the prompt, ignoring fine-grained location and instance count.
• AP50 (primary) — range: percent
  • Average Precision at an Intersection over Union (IoU) threshold of 0.5. For prompts describing multiple instance groups, a Hungarian algorithm matches predicted groups to ground-truth groups to maximize mean IoU before computing precision.

Input / output format

Input: A sequence or horizontally concatenated set of multiple images paired with an open-ended text prompt describing one or more target instances.

Output: A set of bounding boxes for the predicted target instances, optionally assigned to groups corresponding to the prompt's descriptions.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    # predictions & ground_truth: list of dicts with 'image_idx', 'boxes', 'group_id'
    acc = 1.0 if set(p['image_idx'] for p in predictions) == set(g['image_idx'] for g in ground_truth) else 0.0
    pred_groups = group_by_id(predictions)
    gt_groups = group_by_id(ground_truth)
    matched_ious = hungarian_match(pred_groups, gt_groups)
    ap50 = compute_ap(matched_ious, iou_threshold=0.5)
    return acc, ap50

Common pitfalls

• Models frequently fail to assign correct groups, often predicting only a single group for prompts requiring multiple.
• Models tend to generate redundant low-confidence bounding boxes on irrelevant/negative images, which artificially inflates localization but severely degrades Accuracy (Acc).
• Performance drops significantly for small/medium objects and negative samples compared to large objects.

Evidence (verbatim from paper)

Accuracy (Acc) is used to confirm whether the models can correctly identify which images contain the objects indicated by each text prompt, where the instance quantity and fine-grained location information is not considered. We choose average precision (AP50) as the instance-level metric to verify whether the models can locate the target instances with multi-context inputs. For samples where the text prompt describes multiple groups of instances, we first apply Hungarian algorithm to match each predicted group to the most appropriate ground-truth group, ensuring that the mean intersection over union (IoU) across all predictions is maximized.

Citation

@misc{xu2024mcbench,
  title={MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2410.12332}
}

• arXiv: 2410.12332