Portbench Eval

portbench-eval

Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models — Cho et al. (2026) (arXiv:2603.09301, 2026)

What this evaluates

Evaluates large language models' ability to perform quantitative reasoning and structured decision-making in financial portfolio optimization. It probes whether models can correctly apply convex optimization principles under varying constraints and multi-criteria objectives.

Datasets

• PortBench — total 9500; splits: test (9500); repo https://github.com/noahardyx/PortBench

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly selected optimal portfolio choices out of the total number of questions.

Input / output format

Input: A multiple-choice question describing a portfolio optimization scenario, including asset return/risk statistics, a specific investment objective (e.g., minimize variance, maximize Sharpe ratio), constraint settings (e.g., weight bounds, cardinality limits), and four candidate portfolio allocations (one optimal, three distractors).

Output: The model must output the label of the single portfolio choice that optimally satisfies the given objective and constraints.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
return correct / len(gold_labels)

Common pitfalls

• Models may exploit superficial heuristics (e.g., picking highest return) rather than solving the underlying convex optimization, especially when distractors are highly similar.
• Performance on complex objectives like Sharpe ratio maximization drops below 10% because models struggle to integrate multidimensional risk-return trade-offs and constraint handling simultaneously.
• Evaluators should not assume high accuracy on unconstrained problems generalizes to constrained settings, as restrictive bounds significantly widen performance gaps between models.

Evidence (verbatim from paper)

Figure 4 compares the accuracy of LLMs across the five investment objectives. The results show that GPT attains the highest accuracy for risk-based objectives such as minimizing variance and MDD. This outcome suggests that GPT demonstrates a solid structural understanding of risk-related quantitative concepts and is capable of interpreting and applying mathematically defined objectives.

Citation

@misc{cho2026portbench,
  title={Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models},
  author={Cho et al. (2026)},
  year={2026},
  note={arXiv:2603.09301}
}

• arXiv: 2603.09301