Rulereasoner Eval

rulereasoner-eval

RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling — Liu et al. (2025) (arXiv:2506.08672, 2025)

What this evaluates

This evaluation probes a language model's ability to perform rule-based logical reasoning on both in-distribution and out-of-distribution tasks. It measures how well the model can apply explicit and implicit logical rules to derive correct answers under strict exact-match conditions.

Datasets

• BigBench Hard (BBH) — total 750; splits: test (750)
• BigBench Extra Hard (BBEH) — total 400; splits: test (400)
• ProverQA — total 1500; splits: test (1500)

Metrics

• pass@1 (hard exact match) (primary) — range: percent
  • Algebraic mean of hard exact match across all test instances. Equivalent to pass@1 accuracy under strict extraction and comparison.

Input / output format

Input: Logical reasoning prompts containing a set of rules, facts, and a question (e.g., 'If X then Y. Fact: X. Question: Y?'). Includes both explicit rule sets and implicit contextual descriptions.

Output: Step-by-step reasoning chain followed by a final answer formatted as 'Answer: [value].' (e.g., 'Answer: True.' or 'Answer: A.')

Scoring recipe

def compute_pass_at_1(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        # Strict extraction: parse the final answer line
        pred_ans = extract_last_answer(pred)
        if pred_ans == gold:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Using micro-averaged accuracy instead of the specified algebraic (macro) mean across tasks/domains.
• Failing to apply strict extraction for the final answer, which can cause valid reasoning chains to be marked incorrect if the answer format deviates.
• Confusing in-distribution (ID) and out-of-distribution (OOD) evaluation splits, as the paper reports separate results for each.

Evidence (verbatim from paper)

Evaluation Metrics. All tasks in the work are evaluated using the algebraic mean of hard exact match, which is also equivalent to pass@1 accuracy under strict extraction and comparison.

Citation

@misc{liu2025rulereasoner,
  title={RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.08672}
}

• arXiv: 2506.08672