Onethinker Eval

onethinker-eval

OneThinker: All-in-one Reasoning Model for Image and Video — Feng et al. (2025) (arXiv:2512.03043, 2025)

What this evaluates

Evaluates a unified multimodal reasoning model's ability to perform visual understanding tasks across both static images and videos. It probes capabilities in question answering, captioning, spatial and temporal grounding, object tracking, and segmentation.

Datasets

• MMMU — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• MathVerse — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• ScienceQA — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• MMT-Bench — total ?; splits: test (-1)
• VideoMMMU — total ?; splits: test (-1)
• MMVU — total ?; splits: test (-1)
• VideoMME — total ?; splits: test (-1)
• VideoHolmes — total ?; splits: test (-1)
• LongVideoBench — total ?; splits: test (-1)
• LongVideo-Reason — total ?; splits: test (-1)
• VideoMathQA — total ?; splits: test (-1)
• MMSci-Caption — total ?; splits: test (-1)
• MMT-Caption — total ?; splits: test (-1)
• VideoMMLU-Caption — total ?; splits: test (-1)
• Charades — total ?; splits: test (-1)
• ActivityNet — total ?; splits: test (-1)
• ANet-RTL — total ?; splits: test (-1)
• RefCOCO — total ?; splits: testA (-1), testB (-1), val (-1)
• RefCOCO+ — total ?; splits: testA (-1), testB (-1), val (-1)
• RefCOCOg — total ?; splits: test (-1), val (-1)
• STVG — total ?; splits: test (-1)
• GOT-10k — total ?; splits: test (-1)
• MeViS — total ?; splits: test (-1)
• ReasonVOS — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions or matched captions against ground truth references.
• IoU / J&F / R@k / AO — range: [0, 1]
  • Intersection over Union (IoU) for grounding/segmentation, Jaccard (J) and F-score (F) for video segmentation, Recall@k (R@k) for temporal grounding, and Average Overlap (AO) for tracking.

Input / output format

Input: Image or video frames accompanied by a text prompt (question, referring expression, or tracking instruction).

Output: Text response generated via greedy decoding, capped at 4096 tokens.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type in ['qa', 'caption']:
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return (correct / len(golds)) * 100
    elif task_type in ['grounding', 'segmentation']:
        ious = [compute_iou(p, g) for p, g in zip(predictions, golds)]
        return {'IoU': mean(ious), 'mIoU': mean(ious)}
    elif task_type == 'tracking':
        overlaps = [compute_overlap(p, g) for p, g in zip(predictions, golds)]
        return {'AO': mean(overlaps), 'R@0.3': sum(o >= 0.3 for o in overlaps) / len(overlaps)}

Common pitfalls

• Greedy decoding is used for all benchmarks, which may penalize models on open-ended reasoning or creative captioning tasks compared to sampling-based decoding.
• Frame sampling limits vary; this work caps training at 128 frames, but evaluation benchmarks have different native frame counts, requiring careful resampling or padding.
• RefCOCO benchmarks require strict adherence to official testA/testB/val splits; mixing splits inflates scores and breaks comparability.

Evidence (verbatim from paper)

We evaluate models using greedy decoding, following prior works [44, 9, 45]. ... On ActivityNet, our model attains 65.0 R@0.3, 43.6 R@0.5, and 25.7 R@0.7, confirming superior temporal grounding abilities. ... On GOT-10k, OneThinker reaches a high 73.0 AO, 93.9 R@0.3, 84.4 R@0.5, and 68.8 R@0.7... For video segmentation, it reaches 48.8 J, 56.7 F, and 52.7 J&F on MeViS...

Citation

@misc{feng2025onethinker,
  title={OneThinker: All-in-one Reasoning Model for Image and Video},
  author={Feng et al. (2025)},
  year={2025},
  note={arXiv:2512.03043}
}

• arXiv: 2512.03043