Cs 4k Eval

cs-4k-eval

ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows — Wang et al. (2025) (arXiv:2510.20279, 2025)

What this evaluates

Evaluates LLMs on end-to-end computer science research workflows by testing their ability to answer scientific questions grounded in academic papers. It probes domain-specific reasoning, factual recall, and methodological understanding across eight research workflow categories.

Datasets

• CS-4k — total 4000; splits: test (4000); repo https://github.com/wph6/ResearchGPT

Metrics

• model response score (primary) — range: [0, 1]
  • A correctness score computed by querying an LLM judge or rule-based checker on whether the model's response matches the ground-truth answer. During dataset construction, this score is used to filter out trivial or ambiguous questions; for benchmarking, it serves as the primary accuracy metric.

Input / output format

Input: A natural language question grounded in a specific computer science research topic, typically accompanied by retrieved context or paper excerpts from the source arXiv documents.

Output: A natural language answer addressing the question, expected to be factually grounded in the source paper content.

Scoring recipe

def compute_model_response_score(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # LLM-judge or exact-match check for correctness
        if is_correct(pred, gold):
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Ground truth answers are LLM-generated and filtered, so minor hallucinations or phrasing differences may exist despite quality control.
• The test set (CS-4k) is a small, manually curated subset (~4k) of a larger filtered pool, which may not fully represent the difficulty or category distribution of the full corpus.
• Difficulty scores are estimated by an LLM-based scorer, which can introduce systematic bias in how 'hard' questions are balanced across splits.

Evidence (verbatim from paper)

For each Q&A pair, we compute the model response score of these models’ responses, using a prompt (Appendix[A.2]) that explicitly queries model correctness. Questions that are consistently answered correctly by all models (too trivial) or consistently answered incorrectly (too difficult or ambiguous) are removed.

Citation

@misc{wang2025researchgpt,
  title={ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.20279}
}

• arXiv: 2510.20279