Papermind Eval

papermind-eval

PAPERMIND: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs — Yanjun Zhao et al. (arXiv:2604.21304, 2026)

What this evaluates

Evaluates multimodal LLMs' ability to perform integrated agentic reasoning and critical assessment over scientific papers. It probes capabilities in multimodal grounding, experimental interpretation, cross-source evidence synthesis via tool use, and critical evaluation of research claims.

Datasets

• PaperMind — total ?; splits: test (-1); repo https://github.com/Yanjun-Zhao/PaperMind

Metrics

• F1 score (primary) — range: [0, 1]
  • Token-level or entity-level F1 score computed as the harmonic mean of precision and recall between the model's generated answer and the ground truth.
• LLM-as-a-Judge — range: other
  • A 5-point scale rating generated by an LLM judge evaluating the quality and correctness of the model's response against the gold answer and context.
• Average interaction steps — range: other
  • Mean number of reasoning steps or turns the model takes to answer a question.
• Average tool usage — range: other
  • Mean number of external tool invocations per question during agentic reasoning tasks.

Input / output format

Input: Multimodal scientific papers (text and figures) paired with task-specific questions. Inputs may optionally include the paper's introduction as background context, and for agentic tasks, the model receives a query requiring external evidence retrieval.

Output: Free-form text answers to questions, and for agentic tasks, structured tool usage traces (invocations and results) generated via the smolagents/ReAct framework.

Scoring recipe

def compute_f1(pred, gold):
    pred_tokens = set(pred.split())
    gold_tokens = set(gold.split())
    tp = len(pred_tokens & gold_tokens)
    fp = len(pred_tokens - gold_tokens)
    fn = len(gold_tokens - pred_tokens)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_llm_judge(pred, gold, ctx):
    prompt = f'Judge answer: {pred} vs gold: {gold} in context: {ctx}'
    return llm_judge_model(prompt) # returns 1-5

Common pitfalls

• Performance is highly sensitive to the maximum tool call budget; exceeding 6-8 steps often degrades performance due to context dilution.
• F1 scores and LLM-as-a-Judge ratings can diverge, meaning high lexical overlap does not guarantee high qualitative judgment scores.
• Models tend to over-rely on general web search rather than domain-specific tools (e.g., arXiv retriever), which may skew cross-domain evaluation results.

Evidence (verbatim from paper)

We evaluate model performance using both F1 score and an LLM-as-a-Judge metric on a 5-point scale.

Citation

@misc{zhao2026papermind,
  title={PAPERMIND: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs},
  author={Yanjun Zhao et al.},
  year={2026},
  note={arXiv:2604.21304}
}

• arXiv: 2604.21304