Visual Reasoning Eval

visual-reasoning-eval

Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models — Tan et al. (2025) (arXiv:2503.20752, 2025)

What this evaluates

Evaluates vision-language models on complex visual reasoning tasks including arithmetic counting, structural perception, and spatial transformations. It specifically probes the model's ability to generalize under domain shifts and adapt to distribution changes with limited data.

Datasets

• CLEVR-Math — total 36000; splits: train (35000), test (1000)
• Super-CLEVR — total 2000; splits: test (2000)
• Geo170K/Math360K/Geometry3K — total 6120; splits: train (4500), test (1620)
• TRANCE — total 66000; splits: train (60000), test (6000)

Metrics

• accuracy-rate (Acc) (primary) — range: percent
  • Calculated as the proportion of correctly answered instances. Correctness is determined by mathematical equivalence for numerical answers, string matching for multiple-choice questions, and stepwise multi-level evaluation for function-type sequences.

Input / output format

Input: Multimodal instances containing 3D scene images (or charts/architectures) paired with text prompts describing arithmetic, structural, or spatial transformation tasks.

Output: Textual Chain-of-Thought reasoning traces followed by a final answer, optionally structured with XML-like tags (, , , ).

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if gold['type'] == 'numerical':
            if math_equivalent(pred['answer'], gold['answer']):
                correct += 1
        elif gold['type'] == 'multiple_choice':
            if string_match(pred['answer'], gold['answer']):
                correct += 1
        elif gold['type'] == 'function_sequence':
            if stepwise_multi_level_eval(pred['sequence'], gold['sequence']):
                correct += 1
    return correct / len(predictions)

Common pitfalls

• Domain shift (DS) evaluation relies on custom subsets (DS-D, DS-M, DS-L, DS-R) rather than standard dataset splits, requiring careful reconstruction or rendering.
• Function-type sequence answers require stepwise multi-level evaluation rather than simple exact match, which can be ambiguous if the stepwise criteria are not strictly defined.
• Format rewards (e.g., enforcing / tags) can artificially inflate early training metrics without improving actual reasoning correctness or generalization.

Evidence (verbatim from paper)

Evaluation Metrics We use accuracy-rate (Acc) as the primary metric [[86]]. For numerical answers, correctness is verified by mathematical equivalence to the ground truth. For multiple-choice questions, we perform a string match. For function-type sequences, we use stepwise multi-level evaluation.

Citation

@misc{tan2025reasonrft,
  title={Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models},
  author={Tan et al. (2025)},
  year={2025},
  note={arXiv:2503.20752}
}

• arXiv: 2503.20752