Handmethat Eval

handmethat-eval

HandMeThat: Human-Robot Communication in Physical and Social Environments — Wan et al. (2023) (arXiv:2310.03779, 2023)

What this evaluates

Evaluates a robot's ability to understand ambiguous human instructions and infer human subgoals in physically and socially complex household environments. It probes pragmatic reasoning, goal recognition, and collaborative task completion under partial observability.

Datasets

• HandMeThat — total 10000; splits: test (1000)

Metrics

• average score — range: other
  • Mean reward per episode, averaged over 1,000 test episodes.
• success rate (primary) — range: percent
  • Percentage of episodes where the agent achieves the goal within a maximum of 40 steps.
• average moves (successful) — range: other
  • Mean number of steps taken to complete the task, averaged only over successful episodes.

Input / output format

Input: Textual instruction/utterance (potentially ambiguous), task description, and environment observation (object states and locations) provided in either fully or partially observable settings.

Output: Discrete action selection from a valid action space (15–30 options per step), typically involving picking/placing objects or asking clarification questions.

Scoring recipe

def evaluate(predictions, gold, max_steps=40):
    successful_moves = []
    total_reward = 0.0
    for pred_seq, episode in zip(predictions, gold):
        total_reward += episode_reward(episode)
        if len(pred_seq) <= max_steps and check_goal_reached(pred_seq, episode):
            successful_moves.append(len(pred_seq))
    success_rate = (len(successful_moves) / len(gold)) * 100
    avg_moves = sum(successful_moves) / len(successful_moves) if successful_moves else 0
    avg_score = total_reward / len(gold)
    return avg_score, success_rate, avg_moves

Common pitfalls

• The Heuristic baseline assumes full observability and ground-truth symbolic utterance representations, making direct comparison with learning models in partial observability settings unfair.
• Level 3 and 4 tasks are intentionally designed to be 'intrinsically unsolvable' due to insufficient contextual information, which can be mistaken for model failure rather than a benchmark feature requiring clarification.
• DRRN baselines fail primarily due to a large action space (15-30 valid actions) and sparse rewards, not necessarily a lack of pragmatic reasoning capability.

Evidence (verbatim from paper)

We consider three evaluation metrics: 1) the average score of the model; 2) the success rate that the model achieves the goal within limited steps (40); 3) the average number of moves of successful episodes. Scores are averaged on 1,000 episodes.

Citation

@misc{wan2023handmethat,
  title={HandMeThat: Human-Robot Communication in Physical and Social Environments},
  author={Wan et al. (2023)},
  year={2023},
  note={arXiv:2310.03779}
}

• arXiv: 2310.03779