qhjqhj00 game-eval
D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI — Suhwan Choi et al. (2025) (arXiv:2510.05684, 2025)
What this evaluates
Evaluates a vision-action model's ability to predict actions and scale ratios from video game footage, testing in-distribution and out-of-distribution generalization across 2D and 3D games.
Datasets
- Video Games (In-Distribution & OOD) — total ?; splits: train (-1), test (-1)
Metrics
Pearson correlation(primary) — range: other- Linear correlation coefficient between predicted and ground-truth action coordinates (X, Y). Measures alignment of predicted trajectories with actual movements.
Keypress Accuracy— range: percent- Percentage of correctly predicted keypresses (keyboard or mouse) out of total steps. Calculated as (number of correct predictions / total steps) * 100.
Input / output format
Input: Video frames and previous actions.
Output: Predicted action keys and scale ratios.
Scoring recipe
pearson_x = np.corrcoef(true_x, pred_x)[0, 1]
pearson_y = np.corrcoef(true_y, pred_y)[0, 1]
correct_keys = sum(1 for p, g in zip(pred_keys, gold_keys) if p == g)
keypress_acc = (correct_keys / len(gold_keys)) * 100
Common pitfalls
- Pearson correlation measures linear alignment but ignores absolute scale, which is why Scale Ratio is also reported.
- Keypress accuracy differs significantly between keyboard and mouse inputs, so aggregating them without distinction can be misleading.
Evidence (verbatim from paper)
We employ an autoregressive inference pipeline to generate actions and evaluate model performance across multiple metrics... our Generalist-IDM achieves strong performance across all environments. Notably, it yields large gains in Pearson correlation (e.g., +39.5 points on Stardew Valley X) and Keyboard accuracy (e.g., +57.6 points on Brotato), demonstrating robust generalization over diverse control dynamics.
Citation
@misc{choi2025d2e,
title={D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI},
author={Suhwan Choi et al. (2025)},
year={2025},
note={arXiv:2510.05684}
}
- arXiv: 2510.05684