Orak Eval

orak-eval

Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games — Park et al. (2025) (arXiv:2506.03610, 2025)

What this evaluates

Evaluates LLM agents' long-horizon decision-making and gameplay capabilities across 12 diverse video games spanning six genres. It probes the effectiveness of different agentic strategies (zero-shot, reflection, planning, skill-management) and the impact of multimodal inputs (text vs. visual) on action inference. The benchmark also assesses generalization to unseen in-game scenarios, out-of-distribution games, and non-game tasks like math and web navigation.

Datasets

• Orak — total 12; splits: test (12); repo https://github.com/krafton-ai/Orak

Metrics

• normalization score (primary) — range: [0, 100] percent
  • The raw game score divided by the maximum possible game score, expressed as a percentage. Average is computed over 3 to 20 independent trials per game.
• Elo rating — range: other
  • Pairwise competitive win rates converted to Elo ratings for two-player games (Street Fighter III, StarCraft II). Agents play multiple rounds against each other under controlled conditions.

Input / output format

Input: Textual game state description, optionally supplemented with visual screenshots. Inputs are provided in three modalities: Text-only, Image-only, or Both.

Output: Discrete game action (e.g., button press, movement command, or menu selection) inferred by the LLM agent at each step.

Scoring recipe

def compute_normalization_score(raw_score, max_score):
    return (raw_score / max_score) * 100

def evaluate_agent(trial_scores, max_score):
    scores = [compute_normalization_score(s, max_score) for s in trial_scores]
    return sum(scores) / len(scores)

# For competitive games:
# Compute pairwise win rates across 3 rounds, then convert to Elo rating.

Common pitfalls

• Using absolute game scores instead of the required normalization score makes cross-game comparison invalid.
• Applying a zero-shot strategy to complex games (e.g., Pokémon, Minecraft) will yield near-zero performance; genre-specific agentic strategies (reflection, planning, skill-management) are required.
• Assuming visual input always improves performance; for narrative-heavy or abstract games where text contains all necessary state, adding images can degrade accuracy.

Evidence (verbatim from paper)

For each game, we report the normalization score rather than the absolute score, i.e., the game score is normalized by the maximum game score. We report the average score of 3 to 20 trials for each game.

Citation

@misc{park2025orak,
  title={Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2506.03610}
}

• arXiv: 2506.03610