Hftbench Eval

hftbench-eval

Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs — Kang et al. (2025) (arXiv:2505.19481, 2025)

What this evaluates

Evaluates an LLM agent's ability to execute profitable high-frequency trading decisions under strict latency constraints, balancing response speed with financial accuracy. The benchmark measures how well the model recognizes market patterns and executes trades within a fixed time window without degrading portfolio performance.

Datasets

• HFTBench — total ?; splits: test (-1); repo https://github.com/HaoKang-Timmy/LatencySensitiveBench

Metrics

• Daily Yield (%) (primary) — range: percent
  • Percentage change in portfolio value over a 1-minute evaluation window, starting with $10,000 initial cash and a profit threshold of 2%. Calculated as (final_cash - 10000) / 10000 * 100.
• Latency (ms) — range: ms
  • End-to-end inference time in milliseconds required to generate a trading decision.

Input / output format

Input: Real-time stock market data (e.g., Nvidia and Amazon prices) and current portfolio state.

Output: Trading action (e.g., buy/sell/hold) and corresponding quantity.

Scoring recipe

cash = 10000
for each 1-min window:
    obs = get_stock_data()
    action = model(obs)
    cash = execute_trade(cash, action)
daily_yield = (cash - 10000) / 10000 * 100
latency = measure_inference_time_ms()

Common pitfalls

• Assuming lower latency always improves yield; the paper shows that faster but inaccurate decisions on smaller models actually increase losses.
• Ignoring the fixed profit threshold (2%) and time window (1 min) when calculating yield, which standardizes the evaluation period across models.

Evidence (verbatim from paper)

Table[1] demonstrates that FPX , by dynamically trading off latency and quality through adaptive model size and bitwidth selection, achieves the highest daily yield on HFTBench and the best overall reward across both benchmarks.

Citation

@misc{kang2025winfast,
  title={Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs},
  author={Kang et al. (2025)},
  year={2025},
  note={arXiv:2505.19481}
}

• arXiv: 2505.19481