Paperqa Eval

paperqa-eval

PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficient — Wang et al. (2026) (arXiv:2601.12988, 2026)

What this evaluates

Evaluates the accuracy and interaction efficiency of LLM agents performing multi-turn tool-use for scientific paper question-answering. It probes the model's ability to plan, execute tool calls, and extract answers from complex academic documents without excessive interaction or getting stuck in loops.

Datasets

• AirQA-Real — total ?; splits: test (-1)
• SciDQA — total ?; splits: test (-1)

Metrics

• Avg. (primary) — range: percent
  • Average accuracy across sub-task modalities (text, table, image, form, meta). Calculated as the mean of correct answers per query.
• I-Avg (primary) — range: percent
  • I-Avg = Avg * sqrt(1 - I_bar / I_max), where I_bar is the average number of interaction turns and I_max is the maximum allowed interaction turns. Jointly measures performance and efficiency.
• Valid Answers — range: percent
  • Proportion of trials that successfully conclude within the maximum interaction limit without entering repetitive, non-productive loops.
• Interaction Turns — range: other
  • Average number of tool calls or interaction steps taken per query before submitting a final answer or hitting the limit.

Input / output format

Input: Scientific paper QA questions requiring multi-turn tool use. Agents receive access to academic documents, tables, images, and forms via tool-use APIs to search and retrieve information.

Output: A sequence of tool-use actions (interactions) followed by a final answer. The protocol tracks the exact number of interaction turns until the answer is submitted or the limit is reached.

Scoring recipe

def compute_metrics(predictions, golds, interaction_turns, max_turns=100):
    accuracies = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
    avg_acc = sum(accuracies) / len(accuracies)
    avg_turns = sum(interaction_turns) / len(interaction_turns)
    i_avg = avg_acc * (1 - (avg_turns / max_turns)) ** 0.5
    valid = sum(1 for t in interaction_turns if t <= max_turns) / len(interaction_turns)
    return {'Avg': avg_acc, 'I-Avg': i_avg, 'Valid Answers': valid, 'Interaction Turns': avg_turns}

Common pitfalls

• I-Avg jointly optimizes accuracy and efficiency; optimizing solely for Avg may lead to inefficient, high-turn agents that score poorly on I-Avg.
• Avg aggregates performance across diverse modalities (text, table, image, form, meta), potentially hiding severe weaknesses in specific data types.
• The knowing-doing gap probe uses a separate UCB bandit task, not the main benchmarks, so results there are not directly comparable to Paper-QA scores.

Evidence (verbatim from paper)

Beyond the default metrics provided with each benchmark, we inspired from (Liu et al., [2025]), introducing an additional evaluation metric, I-Avg, defined as: I-Avg = Avg * sqrt(1 - I_bar / I_max), where I_bar is the average number of interaction turns, and I_max denotes the maximum number of interaction turns. The I-Avg metric is designed to provide a balanced assessment of agent behavior by jointly considering both its performance and efficiency.

Citation

@misc{wang2026paperguide,
  title={PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficient},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2601.12988}
}

• arXiv: 2601.12988