Sifthinker Eval

sifthinker-eval

SIFThinker: Spatially-Aware Image Focus for Visual Reasoning — Chen et al. (2025) (arXiv:2508.06259, 2025)

What this evaluates

Evaluates a model's spatial reasoning, fine-grained visual perception, and 3D spatial grounding capabilities. It also measures self-correction ability through dynamic bounding box refinement and general visual-language understanding across multiple benchmarks.

Datasets

• SpatialBench — total ?; splits: test (-1)
• SAT-Static — total ?; splits: test (-1)
• CV-Bench — total ?; splits: test (-1)
• VisCoT_s — total ?; splits: test (-1)
• V*Bench — total ?; splits: test (-1)
• RefCOCO — total ?; splits: val (-1), test-A (-1), test-B (-1)
• RefCOCO+ — total ?; splits: val (-1), test-A (-1), test-B (-1)
• RefCOCOg — total ?; splits: val-u (-1), test-u (-1)
• OVDEval — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1), dev (-1)
• SEED-Bench — total ?; splits: test (-1)
• VQAv2 — total ?; splits: test-dev (-1)
• POPE — total ?; splits: val (-1)

Metrics

• Top-1 Accuracy@0.5 (primary) — range: percent
  • Percentage of correctly predicted bounding boxes where the Intersection over Union (IoU) with the ground truth exceeds 0.5.
• NMS-AP — range: percent
  • Mean Average Precision computed after Non-Maximum Suppression, evaluating multi-object detection robustness.
• Average F1 score — range: percent
  • Macro-averaged F1 score computed across three predefined categories (e.g., Adversarial, Common, Hateful) on the COCO validation set.
• Accuracy / Score — range: percent
  • Standard exact-match or normalized score depending on the benchmark (e.g., VQAv2, MME, MMBench, SEED-I, VisCoT_s, V*Bench, SpatialBench).

Input / output format

Input: An input image paired with a textual question or prompt requiring spatial reasoning, visual perception, or referring expression comprehension.

Output: A textual response containing the final answer. During the reasoning process, the model may generate interleaved text and bounding box coordinates to represent dynamic attention shifts.

Scoring recipe

def compute_metric(predictions, gold, metric_name):
    if metric_name == 'Top-1 Accuracy@0.5':
        correct = sum(1 for p, g in zip(predictions, gold) if iou(p, g) >= 0.5)
        return correct / len(gold)
    elif metric_name == 'NMS-AP':
        return compute_nms_ap(predictions, gold, iou_thresh=0.5)
    elif metric_name == 'Average F1 score':
        f1s = [f1_score(gold_cat, pred_cat) for cat in ['adversarial', 'common', 'hateful']]
        return sum(f1s) / len(f1s)
    else:
        return exact_match_or_normalized_score(predictions, gold)

Common pitfalls

• Evaluating on different base models (LLaVA-1.5-7B, Bunny-Llama3-8B, Qwen2.5-VL-7B) without explicitly controlling for architectural differences when comparing SOTA methods.
• Using VisCoT_s (scene-related non-planar subsets) instead of the full VisCoT benchmark, which may skew fine-grained perception results.
• Confusing RefCOCO val/test-A/test-B splits with RefCOCO+ and RefCOCOg splits, as they use different partitioning strategies.

Evidence (verbatim from paper)

Specifically, we select two structurally similar tasks—Referring Expression Comprehension (REC) and Open-Vocabulary Detection (OVD)—both of which require the model to generate bounding boxes conditioned on textual descriptions. SIFThinker outperforms all previous generalist models with comparable parameters, achieving an average improvement of 1% to 3% over VisRL (93.08 vs. 91.78 on RefCOCO*(Kazemzadeh et al. [2014]), 85.76 vs. 83.90 on RefCOCO+(Mao et al. [2016]), 90.47 vs. 88.82 on RefCOCOg(Mao et al. [2016])). Moreover, in most of cases, SIFThinker even surpasses previous state-of-the-art specialist models (e.g. Grounding-DINO, UNINEXT). To further assess multi-object grounding performance of our method, we adopt OVDEval(Yao et al. [2023])* with NMS-AP as the evaluation metric.

Citation

@misc{chen2025sifthinker,
  title={SIFThinker: Spatially-Aware Image Focus for Visual Reasoning},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2508.06259}
}

• arXiv: 2508.06259