Fine R1 Eval

fine-r1-eval

Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning — He et al. (2026) (arXiv:2602.07605, 2026)

What this evaluates

Evaluates multi-modal large language models on fine-grained visual recognition (FGVR) tasks across six standard datasets. It probes the model's ability to distinguish visually similar sub-categories in both closed-world (seen categories) and open-world (unseen categories) settings using chain-of-thought reasoning.

Datasets

• CaltechUCSD Bird-200 — total ?; splits: train (-1), test (-1)
• Stanford Car-196 — total ?; splits: train (-1), test (-1)
• Stanford Dog-120 — total ?; splits: train (-1), test (-1)
• Flower-102 — total ?; splits: train (-1), test (-1)
• Oxford-IIIT Pet-37 — total ?; splits: train (-1), test (-1)
• FGVC-Aircraft — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Success rate where the ground-truth sub-category is included in the model's generation. Calculated as the fraction of test examples where the prediction matches or contains the gold label.
• text inclusion — range: percent
  • Strict string matching between the predicted text and the ground-truth sub-category name.
• relative semantic similarity — range: percent
  • SS_relative = max(0, (Sim(c, c*) - Sim(c_hat, c*)) / (1 - Sim(c_hat, c*))), where c is the predicted sub-category, c* is the ground truth, and c_hat is the super-category. Sim denotes cosine similarity from the SigLIP text encoder.

Input / output format

Input: An image of an object and a text prompt asking the model to identify its sub-category from a candidate list (closed-world) or generate a prediction (open-world).

Output: Natural language text containing the predicted sub-category name, optionally preceded by chain-of-thought reasoning.

Scoring recipe

def score_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if g in p)
    return correct / len(golds)

def score_ss_relative(predictions, golds, super_categories):
    sim = lambda a, b: cosine_similarity(text_encoder(a), text_encoder(b))
    scores = []
    for p, g, s in zip(predictions, golds, super_categories):
        num = max(0, sim(p, g) - sim(s, g))
        den = 1 - sim(s, g)
        scores.append(num / den if den > 0 else 0)
    return sum(scores) / len(scores)

Common pitfalls

• Open-world predictions may differ in granularity (e.g., 'Boeing 737' vs 'Boeing 737-200'), requiring semantic similarity rather than exact match.
• Ground truth labels can contain redundant distinguishing phrases (e.g., 'Coupe 2012' in 'Audi A5 Coupe 2012'), complicating strict string matching.
• The relative semantic similarity formula uses the super-category as a baseline to normalize predictions, which is non-standard for simple accuracy metrics.

Evidence (verbatim from paper)

We define success on a single example as whether the ground-truth choice is included in the MLLM generation. We report the success rate of all test examples as the accuracy in the closed-world setting. Since evaluating models in the open-world setting presents additional challenges, as predictions may differ in granularity (e.g., Boeing 737 vs. Boeing 737-200), or ground truth may include redundancy for distinguishing from others (e.g.,“Coupe 2012” in Audi A5 Coupe 2012 and Audi S5 Coupe 2012), we use two complementary metrics: (1) text inclusion (Zhang et al., [2024e]), evaluating strict string matching. (2) relative semantic similarity between the text embeddings of predictions and ground truth calculated by the SigLIP (Zhai et al., [2023]) text encoder.

Citation

@misc{he2026finer1,
  title={Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2602.07605}
}

• arXiv: 2602.07605