Imagenethink250k Eval

imagenethink250k-eval

ImageNet-Think-250K: A Large-Scale Synthetic Dataset for Multimodal Reasoning for Vision Language Models — Chitty-Venkata et al. (2025) (arXiv:2510.01582, 2025)

What this evaluates

Evaluates vision-language models' ability to generate structured, step-by-step reasoning (thinking tokens) and final answers for multimodal inputs. It probes reasoning coherence, logical progression, and alignment with reference synthetic reasoning traces.

Datasets

• ImageNet-Think-250K — total 250000; splits: test (-1)

Metrics

• BERTScore (primary) — range: [0, 1]
  • Computes similarity using contextual embeddings from pre-trained BERT models. Reports F1 score and overall similarity. Normalized to [0,1].
• Sentence-BERT Cosine Similarity — range: [0, 1]
  • Measures semantic similarity in a unified sentence embedding space. Normalized to [0,1].
• ROUGE-1 — range: [0, 1]
  • Measures unigram lexical overlap between generated and reference texts. Normalized to [0,1].
• ROUGE-L — range: [0, 1]
  • Computes the longest common subsequence to account for word order. Normalized to [0,1].
• Jaccard Index — range: [0, 1]
  • Ratio of intersection to union of token sets, normalized for text length variations. Normalized to [0,1].
• Overlap Coefficient — range: [0, 1]
  • Ratio of common tokens to the smaller set size, useful for different length texts. Normalized to [0,1].
• TF-IDF Cosine Similarity — range: [0, 1]
  • Measures similarity using term frequency-inverse document frequency weighting. Normalized to [0,1].
• Cosine Similarity — range: [0, 1]
  • Baseline vector-space comparison using raw term frequencies. Normalized to [0,1].

Input / output format

Input: Input image paired with a custom prompt (identical to dataset generation prompt with minor model-specific instructions).

Output: Structured thinking tokens (step-by-step reasoning) followed by a final answer.

Scoring recipe

# For each example m in dataset (M total):
#   s1 = compute_metric(model_output, reference_1)
#   s2 = compute_metric(model_output, reference_2)
#   m_avg = (s1 + s2) / 2
#   m_max = max(s1, s2)
# Avg_avg = mean(m_avg for all m)
# Max_avg = mean(m_max for all m)
# Report both Avg_avg and Max_avg for each metric.

Common pitfalls

• The dataset uses synthetic reasoning traces from two different VLMs as ground truth, so metrics measure alignment with synthetic references rather than human-annotated gold standards.
• The dual aggregation strategy (Avg of Avg vs Avg of Max) can yield divergent conclusions about model robustness vs peak capability, which readers might overlook if only one is reported.
• Metrics are normalized to [0,1] but represent different linguistic properties (lexical vs semantic vs vector-space), making direct cross-metric comparison misleading without understanding their distinct focuses.

Evidence (verbatim from paper)

We evaluate semantic preservation using embedding-based metrics that capture meaning beyond surface-level word overlap. BERTScore computes similarity using contextual embeddings from pre-trained BERT models. We report both the F1 score and the overall BERTScore, which provide robust measures of semantic similarity that correlate well with human judgments.

Citation

@misc{chittyvenkata2025imagenethink250k,
  title={ImageNet-Think-250K: A Large-Scale Synthetic Dataset for Multimodal Reasoning for Vision Language Models},
  author={Chitty-Venkata et al. (2025)},
  year={2025},
  note={arXiv:2510.01582}
}

• arXiv: 2510.01582