# Behavior1k Eval

> Evaluates embodied AI agents on long-horizon, human-centered manipulation tasks in a realistic physics-based simulation. It probes the agent's ability to plan and execute complex sequences of action primitives (pick, place, navigate, etc.) while handling rigid, articulated, and deformable objects. Use when the user wants to benchmark on BEHAVIOR-1K, or asks about evaluating this task. Reports task success rate.

- Skill: `qhjqhj00/behavior1k-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/behavior1k-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/behavior1k-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/behavior1k-eval

---


# behavior1k-eval

> BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation — Chengshu Li et al. (2024) (arXiv:2403.09227, 2024)

## What this evaluates

Evaluates embodied AI agents on long-horizon, human-centered manipulation tasks in a realistic physics-based simulation. It probes the agent's ability to plan and execute complex sequences of action primitives (pick, place, navigate, etc.) while handling rigid, articulated, and deformable objects.

## Datasets

- **BEHAVIOR-1K** — total 1000; splits: test (-1)

## Metrics

- `task success rate` **(primary)** — range: percent
  - Percentage of trials where the agent successfully completes the specified long-horizon manipulation task within the allowed horizon.
- `distance traveled` — range: other
  - Total Euclidean distance navigated by the robot base during the task, measured in meters.
- `simulated time` — range: other
  - Total simulation time elapsed until task completion or termination, measured in seconds.
- `kinematic object disarrangement` — range: other
  - Total displacement of objects caused by the robot's motion during the task, measured in meters.

## Input / output format

**Input**: RGB-D images, 3D object coordinates, and navigation/LiDAR data in real-world; corresponding OMNIGIBSON state observations and rendered images in simulation.

**Output**: Discrete selection of an action primitive (pick, place, push, navigate, dip, wipe) applied to a target object or location.

## Scoring recipe

```python
def compute_metrics(trials, gold_success):
    success_rate = sum(gold_success) / len(gold_success) * 100
    dist_nav = np.mean([t['dist_nav'] for t in trials])
    sim_time = np.mean([t['sim_time'] for t in trials])
    kin_dis = np.mean([t['kin_dis'] for t in trials])
    return success_rate, dist_nav, sim_time, kin_dis
```

## Common pitfalls

- Simplifying grasping via an 'assistive pick' primitive during training masks the true difficulty of physics-based grasping, causing severe performance drops when evaluated with full physics.
- Long-horizon tasks suffer from credit assignment and vanishing gradients in end-to-end visuomotor policies, requiring action primitives and memory to succeed.
- Sim-to-real transfer is heavily impacted by unmodeled visual discrepancies (camera dynamic range, textures) and navigation inaccuracies, not just policy errors.

## Evidence (verbatim from paper)

> Following the metrics proposed in BEHAVIOR-100 [27], we report the success rate and efficiency metrics (distance traveled, time invested, and disarrangement caused) in Table 2 and 3, and the success score Q in Table A.13 in Appendix.

## Citation

```bibtex
@misc{li2024behavior1k,
  title={BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation},
  author={Chengshu Li et al. (2024)},
  year={2024},
  note={arXiv:2403.09227}
}
```

- arXiv: 2403.09227