Guide Research Idea Eval

guide-research-idea-eval

GUIDE: Towards Scalable Advising for Research Ideas — Liu et al. (2025) (arXiv:2507.08870, 2025)

What this evaluates

This evaluation probes a system's ability to act as a scientific advisor by predicting whether research hypotheses will be accepted at a top-tier AI conference. It measures alignment with expert peer-review decisions using ranking-based precision and recall metrics on a held-out set of conference submissions.

Datasets

• ICLR 2025 Submissions Test Set — total 1000; splits: test (1000)

Metrics

• Top-5% Precision — range: percent
  • Among all the hypotheses with the top-5% highest predicted rating, the proportion that were actually accepted.
• Top-30% Precision (primary) — range: percent
  • Among all the hypotheses with the top-30% highest predicted scores, the proportion that were actually accepted.
• Accept Recall — range: percent
  • Among all the hypotheses that were accepted by ICLR 2025, the proportion that appear within the top 30% predictions.

Input / output format

Input: Hypothesis abstract, claimed contribution, method description, experimental setup, and the ten most relevant literature sections retrieved from a database of ICLR papers (2016–2024).

Output: A predicted rating distribution over 10 classes (used to rank hypotheses) and optional text-based advice/evaluation.

Scoring recipe

import numpy as np
def compute_metrics(predictions, gold):
    n = len(predictions)
    ranked_idx = np.argsort(predictions)[::-1]
    top_5 = ranked_idx[:int(n * 0.05)]
    top_30 = ranked_idx[:int(n * 0.30)]
    top_5_prec = np.mean([gold[i] for i in top_5])
    top_30_prec = np.mean([gold[i] for i in top_30])
    accept_rec = np.mean([gold[i] for i in top_30]) / np.sum(gold)
    return top_5_prec, top_30_prec, accept_rec

Common pitfalls

• Confusing Top-k% Precision (fraction of top-ranked items that are accepted) with Accept Recall (fraction of all accepted items that appear in the top-k%).
• The test set consists of pre-publication ICLR 2025 submissions, so results may not generalize to fully published papers or other conferences.
• Uncertainty filtering changes the effective test set size; precision scores reported under confidence thresholds apply only to the filtered subset, not the full 1,000 papers.

Evidence (verbatim from paper)

To measure the advising system's alignment with human experts, the following metrics are adopted, 1. Top-5% Precision: Among all the hypotheses with the top-5% highest predicted rating, the proportion that were actually accepted. 2. Top- 30% Precision: Among all the hypotheses with the top- 30% highest predicted scores, the proportion that were actually accepted. 3. Accept Recall: Among all the hypotheses that were accepted by ICLR 2025, the proportion that appear within the top 30% predictions.

Citation

@misc{liu2025guide,
  title={GUIDE: Towards Scalable Advising for Research Ideas},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2507.08870}
}

• arXiv: 2507.08870