qhjqhj00 embodiedclaw-eval
EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development — Zhou et al. (2026) (arXiv:2604.13800, 2026)
What this evaluates
Evaluates the efficiency and executability of conversational workflow automation for embodied AI development tasks, including environment synthesis, trajectory collection, and VLA model evaluation.
Datasets
- RoboTwin — total ?; splits: test (-1)
Metrics
average task completion time(primary) — range: minutes- Mean wall-clock time (in minutes) required to complete a development task across participants or runs. Lower values indicate higher efficiency.
task completion rate— range: [0, 1]- Proportion of workflow executions that finish correctly and execute successfully without failure. Calculated as successful completions divided by total attempts.
Input / output format
Input: Natural-language instructions or modification requests for embodied AI development tasks (e.g., environment creation, editing, trajectory collection, model evaluation).
Output: Executable simulation environments, trajectory data in training-ready formats, and model evaluation results.
Scoring recipe
def compute_efficiency(times):
return sum(times) / len(times)
def compute_accuracy(success_flags):
return sum(success_flags) / len(success_flags)
Common pitfalls
- Task completion rate measures end-to-end workflow executability, not standard prediction accuracy.
- Efficiency gains from EmbodiedClaw may come with slightly longer runtimes on some tasks due to tool encapsulation, so runtime alone does not indicate inferior performance.
Evidence (verbatim from paper)
We use average task completion time as the efficiency metric, where lower is better. We further evaluate EmbodiedClaw in terms of task completion rate, which measures whether an embodied development workflow is completed correctly and executed successfully.
Citation
@misc{zhou2026embodiedclaw,
title={EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development},
author={Zhou et al. (2026)},
year={2026},
note={arXiv:2604.13800}
}
- arXiv: 2604.13800