Behavior1k Eval

behavior1k-eval

BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation — Chengshu Li et al. (2024) (arXiv:2403.09227, 2024)

What this evaluates

Evaluates embodied AI agents on long-horizon, human-centered manipulation tasks in a realistic physics-based simulation. It probes the agent's ability to plan and execute complex sequences of action primitives (pick, place, navigate, etc.) while handling rigid, articulated, and deformable objects.

Datasets

• BEHAVIOR-1K — total 1000; splits: test (-1)

Metrics

• task success rate (primary) — range: percent
  • Percentage of trials where the agent successfully completes the specified long-horizon manipulation task within the allowed horizon.
• distance traveled — range: other
  • Total Euclidean distance navigated by the robot base during the task, measured in meters.
• simulated time — range: other
  • Total simulation time elapsed until task completion or termination, measured in seconds.
• kinematic object disarrangement — range: other
  • Total displacement of objects caused by the robot's motion during the task, measured in meters.

Input / output format

Input: RGB-D images, 3D object coordinates, and navigation/LiDAR data in real-world; corresponding OMNIGIBSON state observations and rendered images in simulation.

Output: Discrete selection of an action primitive (pick, place, push, navigate, dip, wipe) applied to a target object or location.

Scoring recipe

def compute_metrics(trials, gold_success):
    success_rate = sum(gold_success) / len(gold_success) * 100
    dist_nav = np.mean([t['dist_nav'] for t in trials])
    sim_time = np.mean([t['sim_time'] for t in trials])
    kin_dis = np.mean([t['kin_dis'] for t in trials])
    return success_rate, dist_nav, sim_time, kin_dis

Common pitfalls

• Simplifying grasping via an 'assistive pick' primitive during training masks the true difficulty of physics-based grasping, causing severe performance drops when evaluated with full physics.
• Long-horizon tasks suffer from credit assignment and vanishing gradients in end-to-end visuomotor policies, requiring action primitives and memory to succeed.
• Sim-to-real transfer is heavily impacted by unmodeled visual discrepancies (camera dynamic range, textures) and navigation inaccuracies, not just policy errors.

Evidence (verbatim from paper)

Following the metrics proposed in BEHAVIOR-100 [27], we report the success rate and efficiency metrics (distance traveled, time invested, and disarrangement caused) in Table 2 and 3, and the success score Q in Table A.13 in Appendix.

Citation

@misc{li2024behavior1k,
  title={BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation},
  author={Chengshu Li et al. (2024)},
  year={2024},
  note={arXiv:2403.09227}
}

• arXiv: 2403.09227