Roundtable Policy Eval

roundtable-policy-eval

Roundtable Policy: Improving Scientific Reasoning and Narratives through Confidence-Weighted Consensus of LLMs — Yao et al. (2025) (arXiv:2509.16839, 2025)

What this evaluates

Evaluates LLMs' scientific reasoning and proposal writing capabilities through a multi-task accuracy benchmark and a rubric-based narrative generation task. It also assesses the stability of consensus methods and the consistency of AI graders across structured and open-ended scientific domains.

Datasets

• MultiTask scientific tasks — total ?; splits: test (-1)
• SingleTask scientific proposal writing — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions across nine heterogeneous scientific subtasks (Mechanics 1/2, Electromagnetism, Optimization, Thermodynamics, Semiconductors, Math, Geometry, Biology).
• rubric-based score — range: percent
  • AI-grader evaluation of scientific proposals across three sections (Background, Methodology, Impact) scored on Creativity, Scientific Rigor, and Logical Coherence. Scores are aggregated per rubric cell.
• Kendall's Tau — range: [-1, 1]
  • Pairwise inter-grader agreement coefficient measuring rank correlation among AI arbitrators to assess grader consistency.

Input / output format

Input: Scientific reasoning prompts (multiple-choice or open-ended) for MultiTask; structured writing prompts for scientific proposals (Background, Methodology, Impact) for SingleTask.

Output: Model-generated answers or full scientific proposal drafts.

Scoring recipe

if task_type == 'MultiTask':
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    accuracy = correct / len(golds) * 100
elif task_type == 'SingleTask':
    scores = []
    for proposal in predictions:
        rubric_scores = ai_grader.evaluate(proposal, rubrics=['Creativity', 'Scientific Rigor', 'Logical Coherence'], sections=['Background', 'Methodology', 'Impact'])
        scores.append(rubric_scores)
    avg_score = mean(scores)
return accuracy or avg_score

Common pitfalls

• Rubric scores are subjective and vary significantly across graders for open-ended tasks (e.g., Creativity, Logical Coherence), requiring aggregation across multiple AI arbitrators to mitigate bias.
• Statistical significance is assessed via Wilcoxon signed-rank test on round-level paired differences, not just point estimates, so variance across evaluation rounds must be tracked.

Evidence (verbatim from paper)

Table 1: Benchmark performance of individual LLMs versus Roundtable Policy (RP). Top: accuracy on nine heterogeneous scientific tasks in the MultiTask. Bottom: rubric-based scores on scientific proposal writing in the SingleTask.

Citation

@misc{yao2025roundtable,
  title={Roundtable Policy: Improving Scientific Reasoning and Narratives through Confidence-Weighted Consensus of LLMs},
  author={Yao et al. (2025)},
  year={2025},
  note={arXiv:2509.16839}
}

• arXiv: 2509.16839