Behavior 1k Eval

behavior-1k-eval

MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoning — Mohammad Mohammadi et al. (2025) (arXiv:2505.03035, 2025)

What this evaluates

This evaluation probes a robot's ability to execute long-horizon, zero-shot rearrangement tasks in unexplored indoor-outdoor environments using grounded language reasoning. It measures how well the system interprets natural language instructions, reasons over 3D scene graphs, and coordinates sequential manipulation actions to satisfy multiple goal conditions.

Datasets

• BEHAVIOR-1K — total ?; splits: test (-1)

Metrics

• Success Rate (SR) (primary) — range: [0, 1]
  • Share of tasks where all goal conditions are completed and the done() function is called correctly at the exact moment of completion.
• Total Task Completion (TTC) — range: [0, 1]
  • Share of tasks where all goal conditions are completed, regardless of whether done() is called correctly.
• Task Progression (TP) — range: [0, 1]
  • Average share of completed goal conditions out of all goal conditions across tasks.
• Relative task progression (rTP) — range: [0, 1]
  • Share of completed goal conditions that correspond to observed task objects, used to penalize indefinite searching in ambiguous instructions.

Input / output format

Input: Natural language task instructions/goal conditions, a structured 3D scene graph with instance-level object annotations, and current visual/semantic perception of the environment.

Output: Sequential subpolicy selections (manipulation actions) and a binary done() call indicating task completion.

Scoring recipe

def score(predictions, gold):
    n = len(predictions)
    sr = sum(1 for p in predictions if p['all_goals_met'] and p['done_called']) / n
    ttc = sum(1 for p in predictions if p['all_goals_met']) / n
    tp = sum(p['completed_goals'] / p['total_goals'] for p in predictions) / n
    rtp = sum(p['completed_observed_goals'] / max(p['completed_goals'], 1) for p in predictions) / n
    return {'SR': sr, 'TTC': ttc, 'TP': tp, 'rTP': rtp}

Common pitfalls

• Confusing Success Rate (SR) with Total Task Completion (TTC): SR strictly requires the model to call done() at the exact moment all goals are met, whereas TTC only checks goal completion.
• Ambiguity in open-ended instructions (e.g., 'throw away all boxes'): Models may search indefinitely; rTP specifically penalizes this by measuring how many completed goals actually match observed objects.
• Scene representation dependency: Performance heavily relies on whether the input scene graph includes spatial relations and instance-level filtering, which baseline methods like BUMBLE lack.

Evidence (verbatim from paper)

We compare the following statistics:

Success Rate (SR): share of tasks with all goal conditions completed and done () called correctly.. Note that this does not require the model call done () correctly after doing so. Total Task Completion (TTC): share of tasks with all goal conditions completed, irrespect. of calling done () correctly. Task Progression (TP): average share of completed goal conditions of all goal conditions.

Relative task progression (rTP): share of completed goal conditions that include observed task objects. Removing a component of ambiguity from tasks such as "throw away all boxes" where it is unclear when to stop searching for further objects.

Citation

@misc{mohammadi2025more,
  title={MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoning},
  author={Mohammad Mohammadi et al. (2025)},
  year={2025},
  note={arXiv:2505.03035}
}

• arXiv: 2505.03035