Resyn Eval

resyn-eval

ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models — He et al. (2026) (arXiv:2602.20117, 2026)

What this evaluates

Evaluates the reasoning capabilities of language models on synthetically generated, code-verifiable tasks. It measures performance on both the custom ReSyn dataset and standard reasoning benchmarks using zero-shot generation with specific sampling parameters.

Datasets

• ReSyn — total 16500; splits: train (16000), val (500)

Metrics

• mean@4 (primary) — range: [0, 1]
  • Pass@k accuracy: the fraction of test instances where at least one of the top-k generated samples is judged correct by the verifier.
• mean@128 — range: [0, 1]
  • Pass@k accuracy calculated over 128 generated samples per instance, used specifically for the AIME 2024 benchmark.

Input / output format

Input: Natural language question Q, optionally prefixed with instructions to structure output using and tags.

Output: Candidate solutions a_i containing intermediate reasoning inside … tags and a final answer inside … tags.

Scoring recipe

def compute_mean_k(predictions, verifiers, k=4):
    correct = 0
    for q, verifier in verifiers:
        answers = extract_answers(predictions[q])
        for ans in answers[:k]:
            if verifier(ans):
                correct += 1
                break
    return correct / len(verifiers)

Common pitfalls

• Confusing the training reward signal (format score × answer score) with the zero-shot evaluation metric (pass@k accuracy).
• Assuming verifiers are always LLM-based; the paper specifies they are implemented in code, though LLM verifiers may occasionally error (assigned 0).
• Overlooking that evaluation is strictly zero-shot with temperature 0.8 and top-p 0.95, not fine-tuned or SFT.

Evidence (verbatim from paper)

Table 1: Evaluation of ReSyn versus the base Instruct model across reasoning and math benchmarks. All benchmarks are evaluated in zero-shot conditions using temperature 0.8 and top-p 0.95 sampling. | Model | #Params | BBH (zero-shot) | BBEH | GSM8K-test | AIME 2024 | | --- | --- | --- | --- | --- | --- | | | | mean@4 | mean@4 | mean@4 | mean@128 |

Citation

@misc{he2026resyn,
  title={ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2602.20117}
}

• arXiv: 2602.20117