Rewardmap Eval

rewardmap-eval

RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning — Feng et al. (2025) (arXiv:2510.02240, 2025)

What this evaluates

Evaluates fine-grained visual reasoning and spatial understanding on structured domains like transit maps. It probes the model's ability to follow complex routes, count stops, and verify true/false statements about visual layouts, while also measuring generalization to broader spatial and chart reasoning benchmarks.

Datasets

• ReasonMap — total 696; splits: test (-1)
• ReasonMap-Plus — total 2570; splits: test (-1)
• SEED-Bench-2-Plus — total ?; splits: test (-1)
• SpatialEval — total ?; splits: test (-1)
• V*Bench — total ?; splits: test (-1)
• HRBench — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)

Metrics

• Weighted Acc. (primary) — range: percent
  • Accuracy adjusted by a difficulty-aware weighting scheme applied to ReasonMap and ReasonMap-Plus test sets. Weights are assigned based on fine-grained difficulty annotations for each question.
• Weighted Map Score — range: other
  • A spatial reasoning metric scoring the correctness of route/path extraction on transit maps, adjusted by difficulty weights.
• Accuracy — range: percent
  • Standard exact-match or binary correctness percentage reported on SEED-Bench-2-Plus, SpatialEval, V*Bench, HRBench, ChartQA, and MMStar.

Input / output format

Input: An image (e.g., transit map, chart, or visual scene) paired with a natural language question or instruction.

Output: A natural language text response containing the answer, route description, or true/false verdict.

Scoring recipe

def compute_weighted_acc(predictions, golds, weights):
    weighted_correct = sum(w for p, g, w in zip(predictions, golds, weights) if p.strip().lower() == g.strip().lower())
    total_weight = sum(weights)
    return (weighted_correct / total_weight) * 100

def compute_standard_acc(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return (correct / len(golds)) * 100

Common pitfalls

• Greedy decoding (temperature=0) is used exclusively for all evaluations, which may disadvantage models that perform better with sampling.
• ReasonMap/Plus metrics use difficulty-aware weighting, so raw accuracy scores cannot be directly compared to the unweighted percentages on other benchmarks.
• Models frequently hallucinate routes or confuse stops on complex maps, which simple accuracy metrics may not fully capture without qualitative inspection.

Evidence (verbatim from paper)

Table 1: Evaluations of reference models and fine-tuned models on ReasonMap and ReasonMap-Plus. ... Weighted Acc. | Weighted Map Score ... We first evaluate on the test sets of ReasonMap and ReasonMap-Plus, and we report the metrics adjusted by the difficulty-aware weighting scheme... All evaluations use greedy decoding (temperature=0).

Citation

@misc{feng2025rewardmap,
  title={RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning},
  author={Feng et al. (2025)},
  year={2025},
  note={arXiv:2510.02240}
}

• arXiv: 2510.02240