Tfq Bench Eval

tfq-bench-eval

MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learning — Zhang et al. (2026) (arXiv:2602.10575, 2026)

What this evaluates

Evaluates a model's ability to understand visual metaphors and image implications by verifying multiple factual and inferential propositions per image. It probes fine-grained visual perception, multi-hop reasoning, and theory of mind through structured true-false questioning.

Datasets

• TFQ-Bench — total 14099; splits: Lite (492), Full (14099)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered True/False questions. Calculated as the number of predictions matching the ground truth label divided by the total number of questions.

Input / output format

Input: An image paired with a single true-false proposition (question).

Output: Structured text containing a reasoning process enclosed in ... tags, followed by a final answer enclosed in ... tags. The final answer must be 'T' or 'F'.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        start = pred.find('<answer>') + len('<answer>')
        end = pred.find('</answer>')
        pred_ans = pred[start:end].strip().upper()
        if pred_ans == gold.strip().upper():
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Models may output reasoning but fail to strictly follow the tag format, causing parsing failures if not handled robustly.
• The benchmark includes both basic visual perception questions and complex metaphorical implication questions; reporting aggregate accuracy without splitting by difficulty or question type may mask specific capability gaps.
• TFQ-Bench-Lite is strictly disjoint from TFQ-Data-Full, so evaluating on the full set without checking for data leakage is critical.

Evidence (verbatim from paper)

The accuracy reward assesses whether the final answer is correct, while the format reward penalizes poorly structured outputs. Formally, given a reasoning trajectory $\tau$, the total reward is defined as:

Citation

@misc{zhang2026metaphorstar,
  title={MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learning},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2602.10575}
}

• arXiv: 2602.10575