Budget AI Researcher Eval

budget-ai-researcher-eval

The Budget AI Researcher and the Power of RAG Chains — Franklin Lee et al. (2025) (arXiv:2506.12317, 2025)

What this evaluates

Evaluates a retrieval-augmented generation framework's ability to synthesize novel, feasible, and interesting research abstracts by combining distant topics from AI conference literature. It probes long-range concept recombination and grounded ideation capabilities.

Datasets

• AI Conference Papers (ICLR, NeurIPS, ICML, ACL, ECCV) — total 1000; splits: test (1000); repo https://github.com/hellojoeAoPS11235/ai-research-agent

Metrics

• Interestingness — range: 1-10
  • Average rating on a 1-10 scale assigned by a ChatGPT-4o judge using few-shot examples, measuring how engaging the generated abstract is.
• Novelty (primary) — range: 1-10
  • Average rating on a 1-10 scale assigned by a ChatGPT-4o judge, measuring the originality and newness of the synthesized research idea.
• Feasibility — range: 1-10
  • Average rating on a 1-10 scale assigned by a ChatGPT-4o judge, measuring the practicality and realizability of the proposed research.
• Similarity to Future Publications — range: [0, 1]
  • Average similarity score from 0 to 1 assigned by Llama 3.1 8B Instant, comparing generated abstracts against 1000 papers from 2024 AI conferences to measure alignment with emerging trends.

Input / output format

Input: Research paper abstracts and texts from selected AI conferences, hierarchical topic lists, and prompts instructing the model to generate a new research abstract combining distant topics.

Output: A structured research paper abstract containing a title and an abstract body, optionally followed by an experimental procedure.

Scoring recipe

def score_abstracts(predictions, gold=None):
    scores = []
    for pred in predictions:
        # LLM judge rates 1-10 based on prompt + reference examples
        rating = call_llm_judge(pred, task="novelty")
        scores.append(rating)
    return sum(scores) / len(scores)

def score_similarity(predictions, gold):
    sim_scores = []
    for pred in predictions:
        sim = call_llm_similarity(pred, context_papers=gold)
        sim_scores.append(sim)
    return sum(sim_scores) / len(sim_scores)

Common pitfalls

• LLM judges may exhibit bias toward familiar baseline models or specific prompt phrasings, inflating or deflating scores.
• High novelty scores often correlate with lower feasibility scores due to the inherent trade-off in recombining distant concepts.
• Knowledge cutoffs differ across baseline models, making direct novelty comparisons sensitive to the temporal distribution of training data.

Evidence (verbatim from paper)

For quantitative evaluation, we evaluate the generated ideas from three perspectives (interestingness, novelty, and feasibility) using one or few-shot learning through ChatGPT-4o using examples and ratings from reference (Lu et al. [2024]). Table[1] shows the average rating of sample abstracts generated from each framework.

Citation

@misc{lee2025budgetai,
  title={The Budget AI Researcher and the Power of RAG Chains},
  author={Franklin Lee et al. (2025)},
  year={2025},
  note={arXiv:2506.12317}
}

• arXiv: 2506.12317