# Behavior 1k Eval

> This evaluation probes a robot's ability to execute long-horizon, zero-shot rearrangement tasks in unexplored indoor-outdoor environments using grounded language reasoning. It measures how well the system interprets natural language instructions, reasons over 3D scene graphs, and coordinates sequential manipulation actions to satisfy multiple goal conditions. Use when the user wants to benchmark on BEHAVIOR-1K, or asks about evaluating this task. Reports Success Rate (SR).

- Skill: `qhjqhj00/behavior-1k-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/behavior-1k-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/behavior-1k-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/behavior-1k-eval

---


# behavior-1k-eval

> MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoning — Mohammad Mohammadi et al. (2025) (arXiv:2505.03035, 2025)

## What this evaluates

This evaluation probes a robot's ability to execute long-horizon, zero-shot rearrangement tasks in unexplored indoor-outdoor environments using grounded language reasoning. It measures how well the system interprets natural language instructions, reasons over 3D scene graphs, and coordinates sequential manipulation actions to satisfy multiple goal conditions.

## Datasets

- **BEHAVIOR-1K** — total ?; splits: test (-1)

## Metrics

- `Success Rate (SR)` **(primary)** — range: [0, 1]
  - Share of tasks where all goal conditions are completed and the `done()` function is called correctly at the exact moment of completion.
- `Total Task Completion (TTC)` — range: [0, 1]
  - Share of tasks where all goal conditions are completed, regardless of whether `done()` is called correctly.
- `Task Progression (TP)` — range: [0, 1]
  - Average share of completed goal conditions out of all goal conditions across tasks.
- `Relative task progression (rTP)` — range: [0, 1]
  - Share of completed goal conditions that correspond to observed task objects, used to penalize indefinite searching in ambiguous instructions.

## Input / output format

**Input**: Natural language task instructions/goal conditions, a structured 3D scene graph with instance-level object annotations, and current visual/semantic perception of the environment.

**Output**: Sequential subpolicy selections (manipulation actions) and a binary `done()` call indicating task completion.

## Scoring recipe

```python
def score(predictions, gold):
    n = len(predictions)
    sr = sum(1 for p in predictions if p['all_goals_met'] and p['done_called']) / n
    ttc = sum(1 for p in predictions if p['all_goals_met']) / n
    tp = sum(p['completed_goals'] / p['total_goals'] for p in predictions) / n
    rtp = sum(p['completed_observed_goals'] / max(p['completed_goals'], 1) for p in predictions) / n
    return {'SR': sr, 'TTC': ttc, 'TP': tp, 'rTP': rtp}
```

## Common pitfalls

- Confusing Success Rate (SR) with Total Task Completion (TTC): SR strictly requires the model to call `done()` at the exact moment all goals are met, whereas TTC only checks goal completion.
- Ambiguity in open-ended instructions (e.g., 'throw away all boxes'): Models may search indefinitely; rTP specifically penalizes this by measuring how many completed goals actually match observed objects.
- Scene representation dependency: Performance heavily relies on whether the input scene graph includes spatial relations and instance-level filtering, which baseline methods like BUMBLE lack.

## Evidence (verbatim from paper)

> We compare the following statistics:

Success Rate (SR): share of tasks with all goal conditions completed and done () called correctly.. Note that this does not require the model call done () correctly after doing so. Total Task Completion (TTC): share of tasks with all goal conditions completed, irrespect. of calling done () correctly. Task Progression (TP): average share of completed goal conditions of all goal conditions.

Relative task progression (rTP): share of completed goal conditions that include observed task objects. Removing a component of ambiguity from tasks such as "throw away all boxes" where it is unclear when to stop searching for further objects.

## Citation

```bibtex
@misc{mohammadi2025more,
  title={MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoning},
  author={Mohammad Mohammadi et al. (2025)},
  year={2025},
  note={arXiv:2505.03035}
}
```

- arXiv: 2505.03035