Tictactoe Manipulation Eval

tictactoe-manipulation-eval

DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation — Wan et al. (2020) (arXiv:2005.02588, 2020)

What this evaluates

Evaluates a robot's ability to perform adversarial interaction and temporal reasoning through a structured pick-and-place board game. It tests the integration of vision-based contour segmentation, game-state reasoning via Minimax, and precise motion planning under hardware constraints.

Datasets

• Yale-CMU-Berkeley objects set — total ?; splits: (unstated)

Metrics

• t_sub (primary) — range: seconds
  • Time in seconds to complete one pick-and-place sub-task, explicitly excluding gripper closing time. The metric is averaged over 10 repeated runs per robot cell.

Input / output format

Input: RGB image of the 3x3 checkerboard and colored cubes; game state for Minimax planning.

Output: Pick and place actions for alternating turns until win or tie.

Scoring recipe

def score_tictactoe(predictions, gold):
    cycle_times = []
    for action in predictions:
        t = measure_time(action) - gripper_close_time
        cycle_times.append(t)
    return sum(cycle_times) / len(cycle_times)

Common pitfalls

• Gripper closing time varies by hardware and must be explicitly subtracted from the reported time metric.
• The paper mistakenly refers to 'bin-clearing task' in the Tic-Tac-Toe section; ensure context matches the board game protocol.

Evidence (verbatim from paper)

The time to complete a pick and place sub-task $t_{sub}$ is recorded as a performance metric. To compare the performance of the robot arm, we purposely exclude the gripper closing time from $t_{sub}$ . We performed the bin-clearing task ten times on each robot cell, and the performance metric is averaged over ten repeated tasks.

Citation

@misc{wan2020deepclaw,
  title={DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation},
  author={Wan et al. (2020)},
  year={2020},
  note={arXiv:2005.02588}
}

• arXiv: 2005.02588