Thinkswitcher Math Eval

thinkswitcher-math-eval

ThinkSwitcher: When to Think Hard, When to Think Fast — Liang et al. (2025) (arXiv:2505.14183, 2025)

What this evaluates

Evaluates a model's ability to dynamically switch between short and long chain-of-thought reasoning modes based on task complexity, balancing mathematical problem-solving accuracy against computational efficiency. It probes whether a single reasoning model can adaptively select concise or elaborate reasoning paths without architectural changes or post-training.

Datasets

• GSM8K — total ?; splits: test (-1)
• MATH-500 — total ?; splits: test (-1)
• AIME24 — total ?; splits: test (-1)
• AIME25 — total ?; splits: test (-1)
• LiveAoPS — total ?; splits: test (-1)
• Omni-MATH-500 — total ?; splits: test (-1)
• OlympiadBench — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Percentage of correctly solved problems out of the total number of test instances. Calculated as correct predictions divided by total questions.
• Token Count — range: tokens
  • Average number of generated tokens per question across the dataset. Reflects computational cost and varies based on the selected short or long CoT mode.
• AUC-AC — range: [0, 1]
  • Area under the curve plotting model accuracy against token efficiency across varying decision thresholds. Quantifies the trade-off performance of the adaptive mechanism.
• nAUC-AC — range: [0, 1]
  • Normalized variant of AUC-AC that measures the advantage gained by the adaptive mechanism over a linear interpolation between short-CoT-only and long-CoT-only performance points.

Input / output format

Input: Natural language math word problems or competition-level math questions.

Output: Chain-of-thought reasoning (either short or long form) followed by a final answer. The model internally selects the reasoning mode based on a learned switcher.

Scoring recipe

def compute_metrics(predictions, gold_answers, token_counts):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if extract_final_answer(p) == g)
    accuracy = correct / len(gold_answers)
    avg_tokens = sum(token_counts) / len(token_counts)
    return {'accuracy': accuracy, 'avg_tokens': avg_tokens}

Common pitfalls

• The 'Avg.' column in Table 1 reports macro-averaged accuracy and token counts across all seven benchmarks, not a dataset-weighted average.
• Training data (MATH train, GSM8K train, historical AIME, Omni-MATH subset) is explicitly excluded from test sets to prevent data leakage; evaluators must strictly use the designated test splits.
• Token count is not a fixed hyperparameter but a per-instance metric that varies with the selected reasoning mode; averaging must be done over all test instances, not per-mode.

Evidence (verbatim from paper)

To evaluate the overall trade-off performance, we use the AUC-AC to quantify how well ThinkSwitcher maintains accuracy across its spectrum of token efficiencies, and its normalized variant nAUC-AC to measure the advantage gained by ThinkSwitcher’s adaptive mechanism over a linear interpolation between SC-Only and LC-Only performance points.

Citation

@misc{liang2025thinkswitcher,
  title={ThinkSwitcher: When to Think Hard, When to Think Fast},
  author={Liang et al. (2025)},
  year={2025},
  note={arXiv:2505.14183}
}

• arXiv: 2505.14183