# Portbench Eval

> Evaluates large language models' ability to perform quantitative reasoning and structured decision-making in financial portfolio optimization. It probes whether models can correctly apply convex optimization principles under varying constraints and multi-criteria objectives. Use when the user wants to benchmark on PortBench, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/portbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/portbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/portbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/portbench-eval

---


# portbench-eval

> Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models — Cho et al. (2026) (arXiv:2603.09301, 2026)

## What this evaluates

Evaluates large language models' ability to perform quantitative reasoning and structured decision-making in financial portfolio optimization. It probes whether models can correctly apply convex optimization principles under varying constraints and multi-criteria objectives.

## Datasets

- **PortBench** — total 9500; splits: test (9500); repo https://github.com/noahardyx/PortBench

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly selected optimal portfolio choices out of the total number of questions.

## Input / output format

**Input**: A multiple-choice question describing a portfolio optimization scenario, including asset return/risk statistics, a specific investment objective (e.g., minimize variance, maximize Sharpe ratio), constraint settings (e.g., weight bounds, cardinality limits), and four candidate portfolio allocations (one optimal, three distractors).

**Output**: The model must output the label of the single portfolio choice that optimally satisfies the given objective and constraints.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
return correct / len(gold_labels)
```

## Common pitfalls

- Models may exploit superficial heuristics (e.g., picking highest return) rather than solving the underlying convex optimization, especially when distractors are highly similar.
- Performance on complex objectives like Sharpe ratio maximization drops below 10% because models struggle to integrate multidimensional risk-return trade-offs and constraint handling simultaneously.
- Evaluators should not assume high accuracy on unconstrained problems generalizes to constrained settings, as restrictive bounds significantly widen performance gaps between models.

## Evidence (verbatim from paper)

> Figure 4 compares the accuracy of LLMs across the five investment objectives. The results show that GPT attains the highest accuracy for risk-based objectives such as minimizing variance and MDD. This outcome suggests that GPT demonstrates a solid structural understanding of risk-related quantitative concepts and is capable of interpreting and applying mathematically defined objectives.

## Citation

```bibtex
@misc{cho2026portbench,
  title={Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models},
  author={Cho et al. (2026)},
  year={2026},
  note={arXiv:2603.09301}
}
```

- arXiv: 2603.09301