qhjqhj00 mcu-minecraft-eval
JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse — Li et al. (2025) (arXiv:2503.16365, 2025)
What this evaluates
Evaluates a model's ability to follow natural language instructions to perform multi-step, spatially grounded tasks in an open-world 3D environment (Minecraft). It specifically probes capabilities in mining, combat, crafting, and smelting under human-like visibility constraints.
Datasets
- MCU Benchmark — total ?; splits: test (-1)
Metrics
success rate(primary) — range: [0, 1]- Calculated as the number of successful task completions divided by the total number of evaluation runs (at least 30 per task). Category-level performance is reported as the average success rate across all tasks within that category.
Input / output format
Input: First-person visual observation (screen frame), natural language task instruction, and environment state. Agent location and inventory stats are hidden to match human player visibility.
Output: Discretized action token from a 51-token vocabulary representing camera movements and keyboard/mouse button presses.
Scoring recipe
def compute_success_rate(successful_runs, total_runs):
return successful_runs / total_runs
def compute_category_avg(task_success_rates):
return sum(task_success_rates) / len(task_success_rates)
Common pitfalls
- Maximum execution steps are fixed per task to match the original MCU benchmark protocol; exceeding them counts as failure.
- Inventory and location information are intentionally hidden during evaluation to ensure fair comparison with human players and prior baselines.
- Success is binary per run; the reported metric is strictly the average success rate across at least 30 independent trials per task.
Evidence (verbatim from paper)
We conduct evaluations using two broad benchmarks: (i) the agent’s capacity to interact with the Minecraft environment to complete tasks; and (ii) vision-language tasks (e.g., question answering, spatial grounding) designed to assess the VLM’s understanding of Minecraft-specific knowledge. For the instruction-following tasks, we adopt the MCU Benchmark(Lin et al., [2023]), focusing on four categories—Mine Blocks, Kill Entities, Craft Items, and Smelt Items—that represent a wide range of typical game-play behaviors in Minecraft. We perform each task at least 30 times and report the success rate per task, as well as the average success rate within each category.
Citation
@misc{li2025jarvisvla,
title={JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse},
author={Li et al. (2025)},
year={2025},
note={arXiv:2503.16365}
}
- arXiv: 2503.16365