# Game Eval

> Evaluates a vision-action model's ability to predict actions and scale ratios from video game footage, testing in-distribution and out-of-distribution generalization across 2D and 3D games. Use when the user wants to benchmark on Video Games (In-Distribution & OOD), or asks about evaluating this task. Reports Pearson correlation.

- Skill: `qhjqhj00/game-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/game-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/game-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/game-eval

---


# game-eval

> D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI — Suhwan Choi et al. (2025) (arXiv:2510.05684, 2025)

## What this evaluates

Evaluates a vision-action model's ability to predict actions and scale ratios from video game footage, testing in-distribution and out-of-distribution generalization across 2D and 3D games.

## Datasets

- **Video Games (In-Distribution & OOD)** — total ?; splits: train (-1), test (-1)

## Metrics

- `Pearson correlation` **(primary)** — range: other
  - Linear correlation coefficient between predicted and ground-truth action coordinates (X, Y). Measures alignment of predicted trajectories with actual movements.
- `Keypress Accuracy` — range: percent
  - Percentage of correctly predicted keypresses (keyboard or mouse) out of total steps. Calculated as (number of correct predictions / total steps) * 100.

## Input / output format

**Input**: Video frames and previous actions.

**Output**: Predicted action keys and scale ratios.

## Scoring recipe

```python
pearson_x = np.corrcoef(true_x, pred_x)[0, 1]
pearson_y = np.corrcoef(true_y, pred_y)[0, 1]
correct_keys = sum(1 for p, g in zip(pred_keys, gold_keys) if p == g)
keypress_acc = (correct_keys / len(gold_keys)) * 100
```

## Common pitfalls

- Pearson correlation measures linear alignment but ignores absolute scale, which is why Scale Ratio is also reported.
- Keypress accuracy differs significantly between keyboard and mouse inputs, so aggregating them without distinction can be misleading.

## Evidence (verbatim from paper)

> We employ an autoregressive inference pipeline to generate actions and evaluate model performance across multiple metrics... our Generalist-IDM achieves strong performance across all environments. Notably, it yields large gains in Pearson correlation (e.g., +39.5 points on Stardew Valley X) and Keyboard accuracy (e.g., +57.6 points on Brotato), demonstrating robust generalization over diverse control dynamics.

## Citation

```bibtex
@misc{choi2025d2e,
  title={D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI},
  author={Suhwan Choi et al. (2025)},
  year={2025},
  note={arXiv:2510.05684}
}
```

- arXiv: 2510.05684