Symbench Eval

symbench-eval

CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance — Chen et al. (2025) (arXiv:2502.04350, 2025)

What this evaluates

Probes an LLM's ability to solve symbolic reasoning and planning tasks by dynamically switching between textual reasoning and code generation. It evaluates robustness on both seen and unseen tasks, as well as the model's generalizability across different architectures and complexity levels.

Datasets

• SymBench — total 3700; splits: test (3700); repo https://github.com/yongchao98/CodeSteer-v1.0

Metrics

• Average Normalized Score (AveNorm) (primary) — range: [0, 100] percent
  • AveNorm_j = (1/N) * sum_{i=1 to N} (s_ij / max(s_i)), where s_ij is the success rate of method j on task i, max(s_i) is the maximum success rate achieved on task i across all methods, and N is the total number of tasks. It normalizes each task's performance relative to the best method on that task, then averages across all tasks.

Input / output format

Input: Natural language task prompt or question (100 samples per task varying in complexity).

Output: Final answer or executable code snippet. Code execution is capped at 30 seconds; exceeding this limit marks the task as failed or returns errors.

Scoring recipe

# For each method j and task i:
# s_ij = (correct_predictions / 100) * 100  # success rate %
# max_s_i = max(s_ij for all methods j on task i)
# norm_score_ij = s_ij / max_s_i if max_s_i > 0 else 0
# AveNorm_j = sum(norm_score_ij for all tasks i) / N

Common pitfalls

• Code execution timeout: Methods that output code are evaluated by executing it with a strict 30-second limit; exceeding it automatically marks the task as failed or returns errors for subsequent turns.
• Normalization denominator: The Average Normalized Score divides each task's success rate by the maximum score achieved on that task across all evaluated methods (not a theoretical 100%), which can disproportionately weight tasks where baselines perform poorly.

Evidence (verbatim from paper)

We utilize success rate as the metric for each task. To compare each method, we calculate the Average Normalized Score over all the tested tasks by the following equation: $$ \operatorname {A v e N o r m} _ {j} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {s _ {i j}}{\max \left(s _ {i}\right)} \tag {2} $$ where $\operatorname{AveNorm}j$ is the Average Normalized Score for method $j$, $s{ij}$ is the score of method $j$ for task $i$, $\max(s_i)$ is the maximum score for task $i$, $N$ is the total number of tasks.

Citation

@misc{chen2025codesteer,
  title={CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2502.04350}
}

• arXiv: 2502.04350