Livexiv Eval

livexiv-eval

LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content — Shabtay et al. (2024) (arXiv:2410.10783, 2024)

What this evaluates

This benchmark evaluates the multi-modal reasoning capabilities of Large Multimodal Models (LMMs) on scientific content scraped from ArXiv papers. It specifically probes visual question answering (VQA) on figures and table question answering (TQA) using multiple-choice formats derived from real-time academic publications.

Datasets

• LiveXiv — total 16328; splits: test (16328); HF LiveXiv/LiveXiv; repo https://github.com/NimrodShabtay/LiveXiv

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted multiple-choice answers. Calculated as the number of instances where the model's predicted letter matches the ground truth letter divided by the total number of questions.

Input / output format

Input: An image (figure or table) paired with a multiple-choice question and a set of answer options labeled with letters.

Output: A single letter corresponding to the chosen answer option.

Scoring recipe

correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / total) * 100

Common pitfalls

• Claude-Sonnet's reported accuracy may be inflated because it participated in the automated question-filtering pipeline, potentially biasing the dataset toward question types it is predisposed to solve.
• The benchmark is dynamic and versioned (v0-v4); results are not static and depend heavily on the specific release version and the generation/filtering models used.
• The efficient evaluation method (re-evaluating only 3-5 models to predict others) relies on Item Response Theory (IRT) and may yield higher prediction errors on smaller or more diverse dataset versions.

Evidence (verbatim from paper)

The output letter is then compared with the ground truth and the accuracy is measured. For ease of assimilation and to obtain insights into what type of data the models flourish at, we provide the results from data generated on tables and figures separately.

Citation

@misc{shabtay2024livexiv,
  title={LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content},
  author={Shabtay et al. (2024)},
  year={2024},
  note={arXiv:2410.10783}
}

• arXiv: 2410.10783