Omniscience Eval

omniscience-eval

OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding — Tao et al. (2026) (arXiv:2602.13758, 2026)

What this evaluates

Evaluates the semantic alignment and factual fidelity of automatically generated scientific image captions. It probes whether dense, context-aware captions can replace visual inputs for downstream reasoning tasks and how well they capture complex scientific figures compared to raw human-written captions.

Datasets

• OmniScience — total ?; splits: test (-1), val (5000)
• AI2D — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MM-MT-Bench — total ?; splits: test (-1)
• MSEarth — total ?; splits: test (-1)

Metrics

• cross-modal relevance score (primary) — range: continuous (higher is better)
  • Computed by Qwen3-VL-Reranker-8B, a single-tower cross-encoder that jointly encodes image-caption pairs and models cross-modal interactions via cross attention to yield instance-level semantic grounding scores.
• LLM-as-a-Judge score — range: [1, 5]
  • Mean of four 1–5 scale dimensions: Language Fluency, Information Consistency, Key Information Accuracy, and Detail Level. Scores from Qwen3-VL-235B-A22B-Thinking and Seed-1.5VL are averaged.
• Caption QA accuracy — range: [0, 1]
  • Accuracy of GPT-4o-mini answering VQA questions when the visual placeholder is replaced by a generated caption. Evaluated on AI2D, MMMU, MM-MT-Bench, and MSEarth.

Input / output format

Input: Image + caption (for reranker); generated caption + reference caption + image (for LLM-Judge); question + generated caption (for Caption QA proxy task).

Output: Scalar relevance score (reranker); 1–5 score per dimension (LLM-Judge); correct/incorrect answer label (Caption QA).

Scoring recipe

def compute_cross_modal_relevance_score(images, captions):
    scores = []
    for img, cap in zip(images, captions):
        score = reranker_model.encode_cross_modal(img, cap)
        scores.append(score)
    return mean(scores)

def compute_llm_judge_score(captions, references, images):
    all_scores = []
    for cap, ref, img in zip(captions, references, images):
        scores = [judge.evaluate(cap, ref, img) for judge in [judge1, judge2]]
        all_scores.append(mean(scores))
    return mean(all_scores)

def compute_caption_qa_accuracy(questions, captions, gold_answers):
    correct = 0
    for q, cap, gold in zip(questions, captions, gold_answers):
        pred = reasoning_engine.answer(q, caption=cap)
        if pred == gold: correct += 1
    return correct / len(questions)

Common pitfalls

• Using dual-tower embedding models (e.g., CLIP) instead of the specified cross-encoder reranker, which fails to capture fine-grained scientific figure details and yields less precise scores.
• Evaluating raw captions without article context, causing referential ambiguity (e.g., 'as shown in the left panel') and artificially low alignment scores compared to self-contained recaptioned text.
• Treating LLM-as-a-Judge scores as absolute ground truth without acknowledging they are relative rankings validated only on a 300-sample human-annotated subset (Kappa 0.831).

Evidence (verbatim from paper)

We employ Qwen3-VL-Reranker-8B to compute fine-grained relevance scores between images and captions. Compared to embedding based dual-tower similarity models, the reranker adopts a single-tower cross-encoder that jointly encodes image–caption pairs and explicitly models cross-modal interactions using cross attention, yielding more precise instance-level semantic grounding.

Citation

@misc{tao2026omniscience,
  title={OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding},
  author={Tao et al. (2026)},
  year={2026},
  note={arXiv:2602.13758}
}

• arXiv: 2602.13758