Reasoning Accuracy Eval

reasoning-accuracy-eval

Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up — Yuan et al. (2024) (arXiv:2410.12323, 2024)

What this evaluates

Evaluates large language models' logical reasoning and problem-solving capabilities across mathematical, algorithmic, and creative tasks. It measures both the correctness of final answers and the computational efficiency of the reasoning process.

Datasets

• Game of 24 — total 1000; splits: test (1000); repo https://www.4nums.com/game/
• BIG-Bench (subset) — total ?; splits: test (-1); HF google/bigbench
• Python Puzzles — total ?; splits: test (-1); repo https://github.com/microsoft/PythonProgrammingPuzzles
• MGSM — total ?; splits: test (-1); repo https://github.com/google-research/url-nlp/tree/main/mgsm
• Shakespearean Sonnet Writing — total ?; splits: test (-1); HF turingmachine/meta-prompting; repo https://github.com/iljones00/Shakespearean-Sonnets-GPT

Metrics

• Acc_logic (primary) — range: percent
  • Percentage of correct responses. An LLM judge (openai-o1) validates the model's final logical reasoning against gold results, returning Correct or Wrong. Accuracy is computed by tallying correct responses over the total number of samples.
• T — range: seconds
  • Average inference time per sample. Calculated as T = (1/N) * sum(T_ij) across all samples, where T_ij is the time taken for the i-th task on the j-th sample.

Input / output format

Input: Task prompt with 1-shot or 2-shot few-shot examples provided via OpenAI API.

Output: Final logical reasoning answer or solution.

Scoring recipe

# Accuracy
acc = 0
for pred, gold in zip(predictions, golds):
    if llm_judge(pred, gold) == "Correct":
        acc += 1
acc_logic = (acc / len(predictions)) * 100

# Efficiency
total_time = sum(times)
t_efficiency = total_time / len(predictions)

Common pitfalls

• Accuracy relies on an LLM-as-a-judge (openai-o1), which may introduce bias or inconsistency compared to exact-match or rule-based grading.
• Efficiency metric includes API network latency and server processing time, not just pure model inference time, making cross-environment comparisons difficult.
• Few-shot settings (1-shot vs 2-shot) drastically change performance, so results must always be reported with the specific setting.

Evidence (verbatim from paper)

Following Suzgun and Kalai ([2024]); Yao et al. ([2024]), we introduce a LLM to validate the final logical reasoning against gold results (Correct/Wrong). We then compute logical accuracy Acc_logic for each logical task by tallying the number of correct responses. Following Yang et al. ([2024a]), we evaluate reasoning efficiency in terms of complexity by calculating T, calculated as the average time spent per task across all samples in the test dataset:

Citation

@misc{yuan2024reversal,
  title={Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up},
  author={Yuan et al. (2024)},
  year={2024},
  note={arXiv:2410.12323}
}

• arXiv: 2410.12323