# Mcu Minecraft Eval

> Evaluates a model's ability to follow natural language instructions to perform multi-step, spatially grounded tasks in an open-world 3D environment (Minecraft). It specifically probes capabilities in mining, combat, crafting, and smelting under human-like visibility constraints. Use when the user wants to benchmark on MCU Benchmark, or asks about evaluating this task. Reports success rate.

- Skill: `qhjqhj00/mcu-minecraft-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mcu-minecraft-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mcu-minecraft-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mcu-minecraft-eval

---


# mcu-minecraft-eval

> JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse — Li et al. (2025) (arXiv:2503.16365, 2025)

## What this evaluates

Evaluates a model's ability to follow natural language instructions to perform multi-step, spatially grounded tasks in an open-world 3D environment (Minecraft). It specifically probes capabilities in mining, combat, crafting, and smelting under human-like visibility constraints.

## Datasets

- **MCU Benchmark** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: [0, 1]
  - Calculated as the number of successful task completions divided by the total number of evaluation runs (at least 30 per task). Category-level performance is reported as the average success rate across all tasks within that category.

## Input / output format

**Input**: First-person visual observation (screen frame), natural language task instruction, and environment state. Agent location and inventory stats are hidden to match human player visibility.

**Output**: Discretized action token from a 51-token vocabulary representing camera movements and keyboard/mouse button presses.

## Scoring recipe

```python
def compute_success_rate(successful_runs, total_runs):
    return successful_runs / total_runs

def compute_category_avg(task_success_rates):
    return sum(task_success_rates) / len(task_success_rates)
```

## Common pitfalls

- Maximum execution steps are fixed per task to match the original MCU benchmark protocol; exceeding them counts as failure.
- Inventory and location information are intentionally hidden during evaluation to ensure fair comparison with human players and prior baselines.
- Success is binary per run; the reported metric is strictly the average success rate across at least 30 independent trials per task.

## Evidence (verbatim from paper)

> We conduct evaluations using two broad benchmarks: (i) the agent’s capacity to interact with the Minecraft environment to complete tasks; and (ii) vision-language tasks (e.g., question answering, spatial grounding) designed to assess the VLM’s understanding of Minecraft-specific knowledge. For the instruction-following tasks, we adopt the MCU Benchmark(Lin et al., [2023]), focusing on four categories—Mine Blocks, Kill Entities, Craft Items, and Smelt Items—that represent a wide range of typical game-play behaviors in Minecraft. We perform each task at least 30 times and report the success rate per task, as well as the average success rate within each category.

## Citation

```bibtex
@misc{li2025jarvisvla,
  title={JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2503.16365}
}
```

- arXiv: 2503.16365