qhjqhj00 redstar-reasoning-eval
RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems? — Xu et al. (2025) (arXiv:2501.11284, 2025)
What this evaluates
Evaluates the model's ability to solve complex mathematical, coding, and general reasoning problems. It probes multi-step reasoning, domain-specific knowledge integration, and long-context handling across diverse benchmarks.
Datasets
- Math & Reasoning Benchmarks — total ?; splits: test (-1); repo https://github.com/QwenLM/Qwen2.5-Math
- Hellobench — total ?; splits: test (-1)
- SedarEval — total ?; splits: test (-1)
- Chinese Graduate Entrance Mathematics Test — total ?; splits: test (-1); repo https://www.chinakaoyan.com/info/article/id/585019.shtml
Metrics
AVG(primary) — range: percent- Percentage of correctly solved problems. Calculated as (number of solved problems / total number of problems) * 100.
Input / output format
Input: Natural language prompts containing mathematical problems, coding tasks, or general reasoning questions, often requiring multi-step solutions.
Output: Step-by-step reasoning traces (Long-CoT) followed by a final answer or solution.
Scoring recipe
def compute_avg(predictions, gold):
correct = 0
for pred, true_ans in zip(predictions, gold):
if extract_final_answer(pred) == true_ans:
correct += 1
return (correct / len(gold)) * 100
Common pitfalls
- Models may output correct reasoning but fail to format the final answer correctly, leading to false negatives in exact-match scoring.
- Long-CoT outputs can exceed context windows or be truncated, causing evaluation scripts to miss the final answer if not properly parsed.
- Aggregating scores across heterogeneous benchmarks without weighting can mask domain-specific weaknesses.
Evidence (verbatim from paper)
All results are verified by human annotaters. “A/B” represents the ratio of the number of problems solved under this type of problem to the total number of problems. AVG represents the ratio of the solved problems under the total problems.
Citation
@misc{xu2025redstar,
title={RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?},
author={Xu et al. (2025)},
year={2025},
note={arXiv:2501.11284}
}
- arXiv: 2501.11284