Real Time Game Playing Eval

real-time-game-playing-eval

Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playing — Yue et al. (2026) (arXiv:2601.04575, 2026)

What this evaluates

Evaluates real-time video game control policies across programmatic and real-game environments, measuring task completion, combat effectiveness, human-like behavior, and instruction-following capability.

Datasets

• Hovercraft — total ?; splits: test (-1)
• Simple-FPS — total ?; splits: test (-1)
• Real Games (DOOM, Quake, Roblox) — total ?; splits: test (-1)

Metrics

• Hovercraft Loop Time (primary) — range: seconds
  • Time in seconds required for the agent to complete a full loop in the Hovercraft environment. Lower values indicate better performance.
• Simple-FPS Combat Score — range: other
  • Number of hits on the enemy minus the number of hits received by the agent. Higher values indicate better performance.
• Human Issue Rate — range: other
  • Sum of normalized counts for six failure modes (wall collisions, shooting into air, missing targets, non-human behavior, idling, camera jitter) divided by video length. Lower values indicate better performance.
• Instruction-Following Success Rate — range: [0, 1]
  • Percentage of runs where the agent successfully completes the maze after receiving a text instruction.
• Keyboard Perplexity — range: [0, 1]
  • Test loss computed as the perplexity of keyboard actions, used to analyze scaling behavior.

Input / output format

Input: RGB game frames (camera feed) + optional text instruction (only provided during instruction-following evaluation)

Output: Discrete keyboard and mouse actions predicted per timestep

Scoring recipe

def score_hovercraft(preds):
    return time_to_complete_loop(preds)

def score_simple_fps(preds):
    return count_hits_on_enemy(preds) - count_hits_received(preds)

def score_human_eval(preds, video_len):
    issues = [count_wall_collisions, count_air_shots, count_missed_targets,
              count_non_human_behavior, count_idle, count_jitter]
    return sum(issues) / video_len

def score_instruction_following(preds, instruction):
    return success_rate(preds, instruction)

def score_test_loss(preds, gold):
    return cross_entropy_perplexity(preds, gold) # keyboard actions only

Common pitfalls

• Hovercraft uses a lower-is-better metric (time), while Simple-FPS uses a higher-is-better metric (combat score).
• Human evaluation issue counts must be normalized by video length; raw counts are not comparable across videos of different durations.
• Text instructions are strictly excluded from programmatic and standard human preference evaluations; they are only used in the instruction-following benchmark.

Evidence (verbatim from paper)

For Hovercraft, we measure the Hovercraft Loop Time (in seconds) required for the agent to complete a full loop. For Simple-FPS, we report the Simple-FPS Combat Score as the number of hits on the enemy minus the number of hits received. ... Human evaluators assessed model quality by counting the occurrences of the following issues during gameplay: (1) colliding with walls; (2) shooting into the air; (3) missing targets (including items or enemies); (4) exhibiting non–human-like behavior (e.g., repeating loops or moving backward); (5) remaining idle; and (6) camera shaking or jittering. ... normalized the counts by the video length. Lower values indicate better performance.

Citation

@misc{yue2026scalingbehaviorcloning,
  title={Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playing},
  author={Yue et al. (2026)},
  year={2026},
  note={arXiv:2601.04575}
}

• arXiv: 2601.04575