Idea2story Eval

idea2story-eval

Idea2Story: An Automated Pipeline for Transforming Research Concepts into Complete Scientific Narratives — Xu et al. (2026) (arXiv:2601.20833, 2026)

What this evaluates

Evaluates a pipeline's ability to transform vague research intents into structured, methodologically grounded research patterns. It probes the system's capacity for methodological abstraction, knowledge graph retrieval, and coherent scientific narrative generation compared to direct LLM prompting.

Datasets

• ICLR & NeurIPS Papers (3-year corpus) — total 13000; splits: corpus (13000); repo https://github.com/AgentAlphaAGI/Idea2Paper.git

Metrics

• LLM-judged preference (novelty, methodological substance, overall research quality) (primary) — range: ordinal
  • An independent LLM (Gemini 3 Pro) blind-compares outputs from Idea2Story and a direct LLM baseline across three dimensions: novelty, methodological substance, and overall research quality. The metric records which system's output is preferred per case.

Input / output format

Input: Underspecified user research intent (e.g., 'I want to build an e-commerce agent that can better understand user intent.')

Output: Structured research pattern containing: Title, Abstract Focus, Problem Definition, Core Research Gap, Method Skeleton, and Innovation Claims.

Scoring recipe

def evaluate_idea2story(predictions, baselines, judge='Gemini 3 Pro'):
    wins = 0
    for pred, base in zip(predictions, baselines):
        prompt = f'Compare for novelty, methodological substance, and overall research quality:\nA: {pred}\nB: {base}\nWhich is better?'
        resp = judge.generate(prompt)
        if 'A' in resp or 'Idea2Story' in resp:
            wins += 1
    return wins / len(predictions)

Common pitfalls

• Relies solely on qualitative LLM-as-a-judge comparisons without statistical significance testing or human evaluation.
• Evaluation covers only a small set of curated cases (3 user ideas), limiting generalizability.
• LLM judge may exhibit inherent biases toward more complex or verbose outputs despite being blinded to the generation method.

Evidence (verbatim from paper)

To reduce evaluation bias, the generated research stories from both approaches are subsequently assessed by an independent large language model (Gemini 3 Pro), which is not involved in either generation process. The evaluator is instructed to compare the outputs in terms of novelty, methodological substance, and overall research quality, without access to the generation method used. Across all evaluated cases, the externally evaluated results consistently favor the outputs generated by Idea2Story.

Citation

@misc{xu2026idea2story,
  title={Idea2Story: An Automated Pipeline for Transforming Research Concepts into Complete Scientific Narratives},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2601.20833}
}

• arXiv: 2601.20833