Streetfighter Eval

streetfighter-eval

Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs — Kang et al. (2025) (arXiv:2505.19481, 2025)

What this evaluates

Tests an LLM agent's real-time decision-making and combat strategy in a video game environment, prioritizing low latency while maintaining competitive win rates. The benchmark probes the model's ability to make timely character actions under a hard frame-rate limit.

Datasets

• StreetFighter — total ?; splits: test (-1); repo https://github.com/HaoKang-Timmy/LatencySensitiveBench

Metrics

• ELO Score (primary) — range: other
  • Standard ELO rating derived from win rates across 40 matches between model pairs. Higher scores indicate better competitive performance.
• Latency (ms) — range: ms
  • Inference time per action in milliseconds.

Input / output format

Input: Game state observations and opponent actions in a fighting game environment.

Output: Character action/move selection.

Scoring recipe

matches = 40
results = []
for i in range(matches):
    state = get_game_state()
    action = model(state)
    outcome = play_action(action)
    results.append(outcome)
elo = compute_elo(results)
latency = measure_inference_time_ms()
# Note: latency < 200ms yields no benefit due to 5 actions/sec game limit

Common pitfalls

• Optimizing for latency below the game's processing limit (~200ms per action), which provides no competitive advantage.
• Prioritizing speed over decision quality; the fastest models (e.g., 1.5B FP8) perform poorly due to limited strategic capability.

Evidence (verbatim from paper)

For the gaming benchmark, we conduct 40 matches between model pairs and compute win rates to derive ElO ratings. ... In StreetFighter, each character action takes a fixed amount of in-game time to complete, with an effective frame rate of around 5 actions per second (i.e., 200ms per action). Any optimization that reduces model latency beyond this threshold yields no further benefit...

Citation

@misc{kang2025winfast,
  title={Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs},
  author={Kang et al. (2025)},
  year={2025},
  note={arXiv:2505.19481}
}

• arXiv: 2505.19481