Curr Reft Eval

curr-reft-eval

Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning — Deng et al. (2025) (arXiv:2503.07065, 2025)

What this evaluates

Evaluates the out-of-domain generalization and reasoning capabilities of vision-language models across visual detection, classification, and multimodal mathematical reasoning tasks, alongside standard multimodal benchmarks.

Datasets

• RefCOCO — total 4000; splits: train (3000), test (1000)
• RefGTA — total 1000; splits: test (1000)
• Pascal-VOC — total 1000; splits: test (1000)
• Math360K — total 4000; splits: train (3000), test (1000)
• CLEVER-70k-Counting — total 500; splits: test (500)
• MathVista — total 6141; splits: test (6141)
• MATH — total 12000; splits: test (12000)
• AI2D — total 5000; splits: test (5000)
• MMBench — total 3000; splits: test (3000)
• MMVet — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• LLaVABench — total 24000; splits: test (24000)

Metrics

• accuracy (primary) — range: percent
  • Correct predictions divided by total test samples. For detection, correctness requires IoU > 0.5 between predicted and ground truth boxes. For classification, exact label match is required. For math, final answer extraction is used.

Input / output format

Input: Image(s) paired with text prompts or questions for tasks including object localization, visual categorization, and multimodal math reasoning.

Output: Text responses containing bounding box coordinates (for detection), class labels (for classification), or step-by-step reasoning and final answers (for math).

Scoring recipe

def compute_accuracy(predictions, golds, task_type):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if task_type == 'detection':
            correct += 1 if iou(pred, gold) > 0.5 else 0
        elif task_type == 'classification':
            correct += 1 if pred == gold else 0
        elif task_type == 'math':
            correct += 1 if extract_final_answer(pred) == gold else 0
    return correct / len(golds) * 100

Common pitfalls

• IoU threshold of 0.5 is strictly required for detection correctness, not just label matching.
• MMVet uses an LLM-based evaluator for unified scoring across 6 sub-tasks, which may introduce evaluator bias.
• Out-of-domain splits (e.g., RefGTA, Pascal-VOC, CLEVER) are evaluated separately from in-domain splits to measure generalization.

Evidence (verbatim from paper)

We use accuracy as unified evaluation metric, defined as correct predictions over total test samples. For detection, a prediction is correct if the IoU between predicted and ground truth boxes exceeds 0.5. In classification, predictions matching ground truth labels are considered correct.

Citation

@misc{deng2025currreft,
  title={Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning},
  author={Deng et al. (2025)},
  year={2025},
  note={arXiv:2503.07065}
}

• arXiv: 2503.07065