Ideation Space Eval

ideation-space-eval

Navigating Ideation Space: Decomposed Conceptual Representations for Positioning Scientific Ideas — Shen et al. (2026) (arXiv:2601.08901, 2026)

What this evaluates

Evaluates a framework's ability to decompose scientific papers into orthogonal conceptual dimensions (problem, method, findings) and model transitions between them. It probes fine-grained conceptual similarity retrieval and assesses whether the model's novelty predictions align with expert human judgments.

Datasets

• ICLR 2025 Submissions — total 500; splits: test (466); repo https://github.com/PLUM-Lab/IdeationSpace
• AI-Researcher — total 93; splits: test (93)

Metrics

• Recall@K (primary) — range: [0, 1]
  • Fraction of ground-truth relevant papers retrieved within the top-K ranked results.
• NDCG@K — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank K, measuring the quality of the ranked list.
• Hit Rate@K — range: [0, 1]
  • Binary indicator: 1 if at least one relevant paper appears in the top-K results, 0 otherwise.
• Pearson correlation — range: [-1, 1]
  • Linear correlation coefficient between the model's predicted novelty scores and expert-annotated novelty scores.
• Spearman correlation — range: [-1, 1]
  • Rank-based correlation coefficient measuring the monotonic relationship between predicted and expert novelty scores.

Input / output format

Input: For retrieval: query vectors extracted from paper components (research problem, method approach, key findings) or concatenated title+abstract text. For novelty: decomposed paper components and retrieved prior work used to compute a novelty score.

Output: For retrieval: a ranked list of candidate papers. For novelty: a scalar novelty score.

Scoring recipe

def compute_recall_at_k(retrieved_top_k, relevant_papers):
    return len(set(retrieved_top_k) & set(relevant_papers)) / len(relevant_papers)

def compute_ndcg_at_k(retrieved_top_k, relevant_papers):
    dcg = sum(1 / log2(i + 2) for i, p in enumerate(retrieved_top_k) if p in relevant_papers)
    idcg = sum(1 / log2(i + 2) for i in range(min(len(relevant_papers), len(retrieved_top_k))))
    return dcg / idcg if idcg > 0 else 0.0

def compute_hit_rate_at_k(retrieved_top_k, relevant_papers):
    return 1.0 if any(p in relevant_papers for p in retrieved_top_k) else 0.0

def compute_correlation(predicted_scores, expert_scores):
    return pearsonr(predicted_scores, expert_scores)[0]

Common pitfalls

• Table K values are scaled by 3x (displayed 9, 30, 90 correspond to actual K=3, 10, 30).
• Retrieval ground truth is constructed via GPT-5.2 processing references and OpenReview reviews, potentially introducing LLM bias.
• Novelty baselines use varying input configurations (title, abstract, concat), making direct comparison sensitive to input length/format.

Evidence (verbatim from paper)

We report Recall@K, NDCG@K, and Hit Rate@K to measure retrieval quality. We measure the correlation between predicted novelty scores and ground-truth expert scores using both Pearson and Spearman correlation coefficients.

Citation

@misc{shen2026navigating,
  title={Navigating Ideation Space: Decomposed Conceptual Representations for Positioning Scientific Ideas},
  author={Shen et al. (2026)},
  year={2026},
  note={arXiv:2601.08901}
}

• arXiv: 2601.08901