Investorbench Eval

investorbench-eval

INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent — Li et al. (2024) (arXiv:2412.18174, 2024)

What this evaluates

Evaluates the sequential financial decision-making capabilities of LLM-based agents across stock, cryptocurrency, and ETF trading environments. It probes the model's ability to process multi-modal market data, manage portfolio risk, and adapt to volatile market conditions over time.

Datasets

• INVESTORBENCH — total ?; splits: warm-up (-1), test (-1)

Metrics

• CR (Cumulative Return) — range: percent
  • Percentage change in portfolio value over the test period: (Final Value / Initial Value - 1) × 100%. Higher is better.
• SR (Sharpe Ratio) (primary) — range: other
  • Mean of periodic returns divided by their standard deviation. Used as the primary tie-breaker when median metrics conflict across epochs.
• AV (Average Drawdown) — range: percent
  • Average of periodic drawdowns over the test period. Lower is better.
• MDD (Maximum Drawdown) — range: percent
  • Maximum peak-to-trough decline in portfolio value over the test period. Lower is better.

Input / output format

Input: Multi-modal market environment state including historical price data, current holdings, portfolio value, and self-reflection outcomes from previous steps, provided as sequential prompts to the LLM agent.

Output: Sequential trading actions per time step (e.g., buy, sell, hold, or specific asset allocation weights) generated by the LLM agent.

Scoring recipe

def compute_metrics(portfolio_values):
    returns = np.diff(portfolio_values) / portfolio_values[:-1]
    cr = (portfolio_values[-1] / portfolio_values[0] - 1) * 100
    sr = np.mean(returns) / np.std(returns) if np.std(returns) > 0 else 0
    drawdowns = (portfolio_values / np.maximum.accumulate(portfolio_values)) - 1
    av = np.mean(drawdowns) * 100
    mdd = np.min(drawdowns) * 100
    return cr, sr, av, mdd
# Report median across 5 repeated epochs per model

Common pitfalls

• The warm-up period is strictly for agent memory/context initialization and must not be included in metric calculations.
• Results are averaged over 5 repeated epochs due to LLM stochasticity; reporting a single run will yield unreliable rankings.
• SR is used as the primary tie-breaker when median metrics conflict across epochs, not CR or MDD.

Evidence (verbatim from paper)

The performance metrics are reported for the test trajectory with the median CR, SR, AV, and MDD from five repeated epochs. (If the median of these metrics does not belong to the same epoch, the performance is based on the trajectory with the median SR.)

Citation

@misc{li2024investorbench,
  title={INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2412.18174}
}

• arXiv: 2412.18174