Synlogic Eval

synlogic-eval

SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond — Liu et al. (2025) (arXiv:2505.19641, 2025)

What this evaluates

Evaluates the logical reasoning and cross-domain generalization capabilities of models trained with verifiable synthetic data. It measures accuracy on mathematical, coding, and logical reasoning benchmarks using a multi-sample generation and verification protocol.

Datasets

• MATH 500 — total ?; splits: test (-1)
• AIME 2024 — total ?; splits: test (-1)
• AMC 2023 — total ?; splits: test (-1)
• LiveCodeBench — total ?; splits: test (-1)
• SynLogic coding validation split — total ?; splits: val (-1)

Metrics

• avg@8 (primary) — range: percent
  • Computes the fraction of correctly solved problems when generating up to 8 independent responses per prompt and accepting if at least one is correct. Reported as a percentage.

Input / output format

Input: Standard prompt template for mathematical, coding, or logical reasoning problems.

Output: Model generates up to 8 independent responses per prompt; correctness is determined by rule-based or symbolic verification.

Scoring recipe

def compute_avg_at_8(predictions, gold):
    correct_count = 0
    for i in range(len(predictions)):
        # predictions[i] contains up to 8 generated responses
        if any(verify_response(resp, gold[i]) for resp in predictions[i]):
            correct_count += 1
    return (correct_count / len(predictions)) * 100

Common pitfalls

• The paper uses avg@8 scoring rather than avg@1 or majority voting, which can significantly inflate reported accuracy compared to single-sample baselines.
• Evaluations are performed on held-out standard benchmarks rather than the synthetic training data itself, so performance on the training distribution is not reported.

Evidence (verbatim from paper)

Figure[10]: Comparative accuracy (%) of models trained with mixed data and math-only data across three mathematical benchmarks: MATH 500, AIME 2024, and AMC 2023. All evaluations of the figure use avg@8 scoring.

Citation

@misc{liu2025synlogic,
  title={SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.19641}
}

• arXiv: 2505.19641