Innoeval Eval

innoeval-eval

InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem — Qiao et al. (2026) (arXiv:2602.14367, 2026)

What this evaluates

Evaluates an AI system's ability to assess scientific research ideas across classification, selection, ranking, and comparison tasks. It probes knowledge-grounded reasoning and multi-perspective evaluation aligned with human expert judgments and conference acceptance standards.

Datasets

• D_point — total 217; splits: test (217); repo https://github.com/zjunlp/InnoEval
• D_group — total 172; splits: test (172); repo https://github.com/zjunlp/InnoEval
• D_pair — total 372; splits: test (372); repo https://github.com/zjunlp/InnoEval

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of correct predictions over total instances. Computed separately for classification, best selection, and pair-wise tasks.
• macro F1 — range: [0, 1]
  • Unweighted mean of recall (or precision) calculated for each class independently, then averaged across all classes. Used for binary and ternary classification tasks.
• Longest Increasing Subsequence Match — range: [0, 1]
  • Length of the longest increasing subsequence of predicted ranks that matches the ground truth order, normalized by the total number of items in the group.

Input / output format

Input: Research idea text extracted from peer-reviewed papers. For group-wise tasks, a set of ideas on similar topics. For pair-wise tasks, two ideas with adjacent or distinct acceptance labels.

Output: For classification: predicted class label (Reject, Poster, Spotlight, Oral, or Highlight). For group-wise: selected best idea or a full ranked list of ideas. For pair-wise: the preferred idea.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    # Accuracy for classification, selection, and pair-wise
    acc = sum(p == g for p, g in zip(predictions, ground_truth)) / len(ground_truth)
    
    # Macro F1 for classification tasks
    classes = list(set(ground_truth))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, ground_truth) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, ground_truth) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, ground_truth) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    
    # LIS Match for ranking tasks
    lis_len = longest_increasing_subsequence_length(ground_truth, predictions)
    lis_match = lis_len / len(ground_truth)
    
    return {'accuracy': acc, 'macro_f1': macro_f1, 'lis_match': lis_match}

Common pitfalls

• The 'Longest Increasing Subsequence Match' metric for ranking is non-standard and may be confused with Kendall's tau or Spearman correlation; it specifically measures the longest subsequence of correctly ordered items.
• Stratified sampling from NeurIPS25/ICLR25 final decisions means the dataset reflects specific conference acceptance distributions, which may limit generalization to other venues or journal submissions.
• Pair-wise tasks are split into 'Easy' and 'Hard' based on label gaps; evaluating them together without stratification can mask performance on difficult comparisons where labels are adjacent.

Evidence (verbatim from paper)

We use Accuracy and macro F1 as the evaluation metrics. We evaluate best selection by Accuracy and ranking task via Longest Increasing Subsequence Match and Accuracy. We directly use Accuracy to evaluate pair-wise tasks.

Citation

@misc{qiao2026innoeval,
  title={InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem},
  author={Qiao et al. (2026)},
  year={2026},
  note={arXiv:2602.14367}
}

• arXiv: 2602.14367