Mini Behavior Eval

mini-behavior-eval

Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI — Jin et al. (2023) (arXiv:2310.01824, 2023)

What this evaluates

Probes long-horizon decision-making and multi-state object interaction in a procedurally generated 3D gridworld. Evaluates an agent's ability to plan and execute complex household tasks under sparse and dense reward signals.

Datasets

• Mini-BEHAVIOR — total ?; splits: test (-1); repo https://github.com/StanfordVL/mini_behavior

Metrics

• success_rate (primary) — range: [0, 1]
  • Binary indicator: 1 if the agent completes the specified task within the 1000-step time limit, 0 otherwise.
• normalized_reward — range: [0, 1]
  • Cumulative reward scaled such that 1.0 corresponds to successfully completing the task, used to track progress under dense reward shaping.

Input / output format

Input: 3D gridworld state/observation and natural language task instruction.

Output: Discrete action (e.g., move, pick, place, open, close) executed sequentially within a 1000-step episode limit.

Scoring recipe

def compute_success_rate(predictions, gold):
    successes = 0
    for pred in predictions:
        if pred.completed_task:
            successes += 1
    return successes / len(predictions)

Common pitfalls

• Procedural generation yields infinite task variations, so fixed test splits must be explicitly sampled or regenerated for reproducibility.
• Sparse reward (+1 only on completion) causes learning failure for medium/hard tasks within 1M steps, necessitating dense reward shaping for meaningful evaluation.
• Fixed 1000-step time limit may truncate valid but longer successful trajectories.

Evidence (verbatim from paper)

We first examine these tasks under the sparse reward setting, where the agent will only receive a reward of +1 if it successfully completes the task. ... Notice that we normalize the reward in the plots such that a reward of 1 would correspond to successfully completing the task.

Citation

@misc{jin2023minibehavior,
  title={Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI},
  author={Jin et al. (2023)},
  year={2023},
  note={arXiv:2310.01824}
}

• arXiv: 2310.01824