Overcooked AI Adaptation Eval

overcooked-ai-adaptation-eval

Benchmark Real-time Adaptation and Communication Capabilities of Embodied Agent in Collaborative Scenarios — Liu et al. (2024) (arXiv:2412.00435, 2024)

What this evaluates

This benchmark evaluates the real-time adaptability and communication capabilities of LLM-powered embodied agents in human-robot collaboration. It probes how well agents adjust their high-level subtask planning and low-level movement paths when faced with dynamic, constrained environments and non-adaptive human partners.

Datasets

• Enhanced Overcooked-AI — total ?; splits: test (-1)

Metrics

• overall score (primary) — range: other
  • Total soup orders served within a fixed time threshold, aggregating subtask and path adaptation performance.
• success rate — range: percent
  • Percentage of short-horizon path adaptation scenarios completed within the timestep limit.
• stuck time — range: other
  • Number of timesteps an agent remains stationary or fails to progress toward its target during path adaptation tests.
• subtask goal accuracy — range: percent
  • Match rate between the agent's generated subtask goal location and the human-labeled ground truth.

Input / output format

Input: Grid-based environment state (layout topology, item positions, agent/human coordinates, timer), assigned subtask, starting frame, and human collaborator actions/trajectory.

Output: Sequence of atomic actions (up, down, left, right, stay, interact) and/or language-based adaptation plans/messages sent to the human collaborator.

Scoring recipe

def score_overcooked(predictions, gold):
    overall_score = count_served_soups(predictions, time_limit=gold.time_limit)
    path_success = sum(1 for s in predictions if s.completed_within_limit) / len(predictions)
    stuck_time = sum(s.timesteps_stuck for s in predictions)
    subtask_acc = match_rate(predictions.subtask_goals, gold.subtask_goals)
    return overall_score, path_success, stuck_time, subtask_acc

Common pitfalls

• Confusing the aggregated overall score with isolated path or subtask adaptation metrics.
• Assuming the environment is fully observable or static; the benchmark explicitly tests adaptation to non-adaptive or unpredictable human partners in confined layouts.
• Overlooking the qualitative evaluation of language-based adaptation plans, which requires human expert comparison rather than automated exact-match scoring.

Evidence (verbatim from paper)

For the quantitative evaluation, we have the agent start on the designed start frame, and the scenario is counted as successfully finished if both agents can complete their assigned subtasks within the limited timesteps. The success rate and stuck time on different scenarios provide us with a direct assessment of the ability in path adaption and spatial reasoning.

Citation

@misc{liu2024overcooked,
  title={Benchmark Real-time Adaptation and Communication Capabilities of Embodied Agent in Collaborative Scenarios},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2412.00435}
}

• arXiv: 2412.00435