Scalar Eval

scalar-eval

SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning — Wang et al. (2025) (arXiv:2502.13753, 2025)

What this evaluates

Evaluates long-context academic reasoning by testing whether LLMs can correctly identify masked citations within scientific papers. It probes the model's ability to understand semantic context, attributional claims, and descriptive references across varying context lengths and difficulty levels.

Datasets

• SCALAR — total ?; splits: easy (-1), hard (-1); repo https://github.com/LibrAIResearch/scalar

Metrics

• accuracy (primary) — range: [0, 1]
  • Accuracy is calculated as the number of correctly identified masked citations divided by the total number of questions. It measures the model's ability to select the ground-truth citation from a set of candidates.

Input / output format

Input: A cloze-style multiple-choice question containing a masked citation within a structured section of a scientific paper. The context provided to the model varies between full paper and section-only, and candidate options vary between full paper metadata and title/abstract-only.

Output: A single selected candidate citation from the provided multiple-choice options.

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return correct / len(gold_labels)

Common pitfalls

• Models may exploit shortcuts by matching titles or abstracts rather than performing deep semantic reasoning about the citation context.
• Performance does not scale linearly with model size; smaller models optimized for long-context windows can outperform larger ones, making size a misleading proxy for capability.
• Difficulty is controlled by four dimensions (paper scope, citation type, distractor sampling, candidate representation), so results can vary significantly depending on which combination is used.

Evidence (verbatim from paper)

On the easy level, SCALARcan already differentiate LLMs’ long context capability, where the best model GPT-4o achieves 95% accuracy, while the lowest-performing models hover around 30-37%, compared to the random baseline of 25%. The hard level proves particularly challenging - even state-of-the-art models like GPT-4o achieve only 50% accuracy, while most other models perform near random chance.

Citation

@misc{wang2025scalar,
  title={SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.13753}
}

• arXiv: 2502.13753