Embodiedgpt Control Eval

embodiedgpt-control-eval

EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought — Yao Mu et al. (2023) (arXiv:2305.15021, 2023)

What this evaluates

This evaluation probes the model's ability to generate executable sub-goal plans from visual inputs and translate them into low-level control actions in simulated robotic environments. It specifically tests closed-loop planning and few-shot policy adaptation across standard embodied AI benchmarks.

Datasets

• Franka Kitchen — total ?; splits: test (-1)
• Meta-World — total ?; splits: test (-1)

Metrics

• success rate (primary) — range: percent
  • Average success rate computed over 100 random evaluations per task, aggregated across 5 tasks per benchmark, 5 random seeds, and 2 camera views.

Input / output format

Input: Visual observations from two camera views, combined with a few-shot demonstration set (10 or 25 examples) for policy learning.

Output: Low-level control actions executed in the simulation environment.

Scoring recipe

total_successes = 0
total_trials = 0
for seed in range(5):
    for camera in range(2):
        for task in tasks:
            for _ in range(100):
                obs = get_observation(task, camera, seed)
                action = policy_network(obs, demonstrations)
                if environment_step(action):
                    total_successes += 1
                total_trials += 1
return (total_successes / total_trials) * 100

Common pitfalls

• The policy network is adapted using few-shot demonstrations (10 or 25), not evaluated zero-shot.
• The reported success rate is an average over 5 seeds, 2 camera views, and 5 tasks; single-run scores are not directly comparable.

Evidence (verbatim from paper)

In all experiments, the policy network is learned using few-shot learning on a small amount of demonstration data. There are two settings, one of which utilizes 10 demonstrations, and the other utilizes 25 demonstrations. We report the success rate in 100 random evaluations with only visual observations in 5 tasks per benchmark over 5 seeds and 2 different camera views for each setting, respectively.

Citation

@misc{mu2023embodiedgpt,
  title={EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought},
  author={Yao Mu et al. (2023)},
  year={2023},
  note={arXiv:2305.15021}
}

• arXiv: 2305.15021