Scitrek Eval

scitrek-eval

Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles — Li et al. (2025) (arXiv:2509.21028, 2025)

What this evaluates

Evaluates long-context language models' ability to perform numerical aggregation, filtering, sorting, and logical operations across extended contexts (up to 1M tokens) using scientific article metadata and full-text articles.

Datasets

• SciTrek — total ?; splits: train (19543), test (-1); repo https://github.com/oaimli/SciTrek

Metrics

• exact match (primary) — range: [0, 1]
  • Binary score: 1 if the model's predicted answer exactly matches the ground truth string, 0 otherwise. Reported as the average across all instances.
• F1 — range: [0, 1]
  • Token-level F1 score calculated between the predicted answer and the ground truth answer, averaged across instances.

Input / output format

Input: Instruction prompt containing either the full-text scientific article or a corresponding database table as context, followed by a question generated via SQL queries over the article metadata.

Output: Three generated answers per question, expected to be factual items such as specific numbers, author names, or article titles.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1_scores = [token_f1(p, g) for p, g in zip(predictions, golds)]
    return {'exact_match': sum(em_scores) / len(em_scores), 'f1': sum(f1_scores) / len(f1_scores)}

Common pitfalls

• The paper states models generate three answers per question but does not specify how to aggregate or select among them for the final metric calculation.
• Performance is highly sensitive to context length, with significant metric drops observed as input exceeds 128K tokens, making cross-length comparisons unstable.
• Out-of-distribution generalization (e.g., to unseen context lengths, topics, or skills like relational filtering) remains poor even after post-training, which can skew aggregate scores if not stratified.

Evidence (verbatim from paper)

We assess performance using average exact match and F1, as the expected outputs are factual items with minimal variation, such as specific numbers, author names, or article titles.

Citation

@misc{li2025scitrek,
  title={Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2509.21028}
}

• arXiv: 2509.21028