# Papermind Eval

> Evaluates multimodal LLMs' ability to perform integrated agentic reasoning and critical assessment over scientific papers. It probes capabilities in multimodal grounding, experimental interpretation, cross-source evidence synthesis via tool use, and critical evaluation of research claims. Use when the user wants to benchmark on PaperMind, or asks about evaluating this task. Reports F1 score.

- Skill: `qhjqhj00/papermind-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/papermind-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/papermind-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/papermind-eval

---


# papermind-eval

> PAPERMIND: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs — Yanjun Zhao et al. (arXiv:2604.21304, 2026)

## What this evaluates

Evaluates multimodal LLMs' ability to perform integrated agentic reasoning and critical assessment over scientific papers. It probes capabilities in multimodal grounding, experimental interpretation, cross-source evidence synthesis via tool use, and critical evaluation of research claims.

## Datasets

- **PaperMind** — total ?; splits: test (-1); repo https://github.com/Yanjun-Zhao/PaperMind

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Token-level or entity-level F1 score computed as the harmonic mean of precision and recall between the model's generated answer and the ground truth.
- `LLM-as-a-Judge` — range: other
  - A 5-point scale rating generated by an LLM judge evaluating the quality and correctness of the model's response against the gold answer and context.
- `Average interaction steps` — range: other
  - Mean number of reasoning steps or turns the model takes to answer a question.
- `Average tool usage` — range: other
  - Mean number of external tool invocations per question during agentic reasoning tasks.

## Input / output format

**Input**: Multimodal scientific papers (text and figures) paired with task-specific questions. Inputs may optionally include the paper's introduction as background context, and for agentic tasks, the model receives a query requiring external evidence retrieval.

**Output**: Free-form text answers to questions, and for agentic tasks, structured tool usage traces (invocations and results) generated via the smolagents/ReAct framework.

## Scoring recipe

```python
def compute_f1(pred, gold):
    pred_tokens = set(pred.split())
    gold_tokens = set(gold.split())
    tp = len(pred_tokens & gold_tokens)
    fp = len(pred_tokens - gold_tokens)
    fn = len(gold_tokens - pred_tokens)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_llm_judge(pred, gold, ctx):
    prompt = f'Judge answer: {pred} vs gold: {gold} in context: {ctx}'
    return llm_judge_model(prompt) # returns 1-5
```

## Common pitfalls

- Performance is highly sensitive to the maximum tool call budget; exceeding 6-8 steps often degrades performance due to context dilution.
- F1 scores and LLM-as-a-Judge ratings can diverge, meaning high lexical overlap does not guarantee high qualitative judgment scores.
- Models tend to over-rely on general web search rather than domain-specific tools (e.g., arXiv retriever), which may skew cross-domain evaluation results.

## Evidence (verbatim from paper)

> We evaluate model performance using both F1 score and an LLM-as-a-Judge metric on a 5-point scale.

## Citation

```bibtex
@misc{zhao2026papermind,
  title={PAPERMIND: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs},
  author={Yanjun Zhao et al.},
  year={2026},
  note={arXiv:2604.21304}
}
```

- arXiv: 2604.21304