Scientific Ideation Eval

scientific-ideation-eval

Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training — Salimi et al. (2026) (arXiv:2604.16723, 2026)

What this evaluates

Evaluates the ability of LLMs to generate novel, feasible, and effective scientific research ideas given a research question. Probes open-ended scientific reasoning and ideation quality under compute-matched inference budgets.

Datasets

• ICLR 2024 & NeurIPS 2025 — total 80; splits: test (80); repo https://github.com/mnsalimi/Debate-as-Reward-A-Multi-Agent-Reward-System-for-Scientific-Ideation-via-RL-Post-Training

Metrics

• Absolute Novelty (primary) — range: [1, 5]
  • LLM judge (Qwen 2.5 72B) independently rates each generated idea on a fixed integer scale from 1 to 5, where 5 indicates the highest quality. Other absolute metrics (Feasibility, Effectiveness) use the same protocol.
• Pairwise Novelty — range: [1, 5]
  • LLM judge evaluates every unordered pair of ideas twice (swapped order to mitigate position bias). Preferences are mapped to +1, -1, or 0, aggregated per idea, and linearly normalized to the 1–5 range.

Input / output format

Input: Research question (extracted from paper abstract/full text or provided by PhD candidates)

Output: Generated scientific idea/abstract text

Scoring recipe

def score_ideas(predictions, judge_model):
    # Absolute scoring
    abs_scores = [judge_model.rate_1_to_5(idea) for idea in predictions]
    
    # Pairwise scoring
    pairwise_scores = []
    for i, idea_i in enumerate(predictions):
        votes = 0
        for j, idea_j in enumerate(predictions):
            if i == j: continue
            # Compare in both orders to mitigate position bias
            votes += judge_model.compare(idea_i, idea_j)  # +1/-1/0
            votes += judge_model.compare(idea_j, idea_i)  # +1/-1/0
        # Linearly normalize aggregated votes to [1, 5]
        normalized = linear_normalize(votes, min_possible, max_possible, 1, 5)
        pairwise_scores.append(normalized)
    return abs_scores, pairwise_scores

Common pitfalls

• LLM evaluators may be biased by familiarity with post-cutoff papers; the protocol mitigates this by using an evaluator with a matching knowledge cutoff date.
• Position bias in pairwise comparisons must be explicitly mitigated by evaluating each idea pair in both orders.
• A known trade-off exists between novelty and feasibility, causing LLMs to penalize highly novel ideas as less feasible.

Evidence (verbatim from paper)

In the absolute scoring setup, the evaluator assigns each idea an integer score from 1 to 5, with 5 indicating the highest quality. For pairwise comparisons, we present every unordered pair of ideas twice—once in each order—to mitigate position bias. The model’s judgment for each comparison is mapped to a numerical value: +1 if the first idea is preferred, –1 if the second is preferred, and 0 if they are deemed equivalent. We aggregate these values across all comparisons for each idea and linearly normalize the resulting scores to the 1–5 range, where 5 again corresponds to the highest-performing method. Table 2 presents the absolute and pairwise evaluation scores of our model and baselines across three key dimensions: novelty, feasibility, and effectiveness.

Citation

@misc{salimi2026debateasreward,
  title={Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training},
  author={Salimi et al. (2026)},
  year={2026},
  note={arXiv:2604.16723}
}

• arXiv: 2604.16723