qhjqhj00 embrace-3k-eval
EmbRACE-3K: Embodied Reasoning and Action in Complex Environments — Lin et al. (2025) (arXiv:2507.10548, 2025)
What this evaluates
Evaluates embodied vision-language models on step-wise, instruction-driven navigation and interaction in complex, partially observable environments. It probes three core capabilities: active exploration, dynamic spatial-semantic reasoning, and multi-stage goal execution under closed-loop perception-action constraints.
Datasets
- EmbRACE-3K — total 3000; splits: test (-1)
Metrics
success(primary) — range: percent- Percentage of tasks where the agent successfully completes the instruction within the allowed step limit.
Input / output format
Input: Egocentric RGB frames, 6-DoF agent pose, and natural language task instruction.
Output: Discrete action sequence and step-wise natural language reasoning/justification for each action.
Scoring recipe
def compute_success(predictions, gold):
correct = 0
for pred, task in zip(predictions, gold):
if pred.completed and pred.steps <= 32:
correct += 1
return (correct / len(gold)) * 100
Common pitfalls
- Trajectories are strictly filtered to a maximum of 32 steps; evaluating agents that exceed this limit will artificially penalize valid long-horizon strategies.
- Task types vary significantly in difficulty (Basic vs. Multi-stage/Interaction); reporting only aggregate success masks capability gaps in complex reasoning categories.
- Egocentric observations and 6-DoF poses must be temporally aligned; mismatched frame-pose pairs break the closed-loop perception-action evaluation.
Evidence (verbatim from paper)
The dataset enables evaluation of VLMs on three core embodied capabilities—Exploration, Dynamic Spatial-Semantic Reasoning, and Multi-stage Goal Execution—where state-of-the-art models achieve <20% success in zero-shot settings, revealing a critical gap in embodied reasoning.
Citation
@misc{lin2025embrace3k,
title={EmbRACE-3K: Embodied Reasoning and Action in Complex Environments},
author={Lin et al. (2025)},
year={2025},
note={arXiv:2507.10548}
}
- arXiv: 2507.10548