Bwor Eval

bwor-eval

OR-LLM-Agent: Automating Modeling and Solving of Operations Research Optimization Problems with Reasoning LLM — Zhang et al. (2025) (arXiv:2503.10009, 2025)

What this evaluates

Evaluates LLMs' ability to automate operations research problem solving through mathematical modeling, code generation, and solver-based optimization. It probes whether reasoning agents can correctly translate natural language OR problems into executable models and compute optimal solutions.

Datasets

• BWOR — total 82; splits: test (82); repo https://github.com/bwz96sco/or_llm_agent

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of problems where the model's final solution matches the ground-truth optimal solution.

Input / output format

Input: LaTeX-formatted natural language problem descriptions, with tabular data included where applicable.

Output: Mathematical model, solver-based code, and optimal solution.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, ground_truth):
    if pred.solution == gold.solution:
        correct += 1
return correct / len(predictions)

Common pitfalls

• Problems are manually translated from Chinese, so models may struggle with translation artifacts or domain-specific terminology.
• Ground-truth answers combine textbook solutions and domain expert computations, requiring careful verification of the scoring baseline.
• The task requires a multi-stage pipeline (modeling, coding, debugging), so evaluating only the final output may obscure where agent failures occur.

Evidence (verbatim from paper)

We release BWOR, an OR benchmark dataset consisting of 82 problems collected from standard OR textbooks. Ground-truth answers are partially sourced from the textbooks and partially computed by domain experts. Experimental results show that DeepSeek-R1-based OR-LLM-Agent outperforms GPT-o3, Gemini 2.5 Pro, and ORLM by at least 7% in accuracy across OR benchmarks...

Citation

@misc{zhang2025orllmagent,
  title={OR-LLM-Agent: Automating Modeling and Solving of Operations Research Optimization Problems with Reasoning LLM},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.10009}
}

• arXiv: 2503.10009