Mocentric Bench Eval

mocentric-bench-eval

PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding? — Siam et al. (2025) (arXiv:2509.02807, 2025)

What this evaluates

Evaluates whether video multi-modal LLMs genuinely utilize motion cues for pixel-level visual grounding. It specifically probes their ability to distinguish true motion from static fake motion (Motion Existence) and to differentiate forward from reversed motion sequences (Motion Order).

Datasets

• MoCentric-Bench — total ?; splits: test (-1); repo https://github.com/MSiam/PixFoundation-2.0.git

Metrics

• mIoU (primary) — range: [0, 1]
  • Intersection over Union between the predicted segmentation mask and the ground-truth mask. Standard for referring segmentation tasks; the exact metric name is not explicitly stated in the provided excerpt, but mIoU is the conventional evaluation standard for this task.

Input / output format

Input: A video sequence (or a single repeated keyframe, reversed video, or multi-video layout combining original and modified videos) paired with a referring expression (standard or motion-centric).

Output: A pixel-level segmentation mask corresponding to the object described in the referring expression.

Scoring recipe

def compute_mIoU(predictions, gold):
    ious = []
    for pred_mask, gt_mask in zip(predictions, gold):
        intersection = np.logical_and(pred_mask, gt_mask).sum()
        union = np.logical_or(pred_mask, gt_mask).sum()
        ious.append(intersection / union if union > 0 else 0.0)
    return np.mean(ious)

Common pitfalls

• Models often rely on static appearance cues (object type, heading, position) rather than temporal dynamics, as many motion expressions can be resolved from a single frame.
• Evaluating only on standard referring expressions without motion-centric probes fails to reveal the model's true spatiotemporal reasoning capabilities.
• Multi-video layout evaluations require careful alignment to prevent spatial confusion during grounding.

Evidence (verbatim from paper)

We focus on referring video segmentation and emphasize both standard and motion referring expressions. We argue that the majority of referring expressions can be identified using strong single-image baselines that do not have an understanding of temporal information. In the motion existence probe, the automatic selection of the keyframe is the crucial step to approximate the motion expression with only one static frame. In the motion order probe, the crucial step is to reverse the motion referring expression to match the reversal of the video.

Citation

@misc{siam2025pixfoundation2,
  title={PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?},
  author={Siam et al. (2025)},
  year={2025},
  note={arXiv:2509.02807}
}

• arXiv: 2509.02807