# Tictactoe Manipulation Eval

> Evaluates a robot's ability to perform adversarial interaction and temporal reasoning through a structured pick-and-place board game. It tests the integration of vision-based contour segmentation, game-state reasoning via Minimax, and precise motion planning under hardware constraints. Use when the user wants to benchmark on Yale-CMU-Berkeley objects set, or asks about evaluating this task. Reports t_sub.

- Skill: `qhjqhj00/tictactoe-manipulation-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tictactoe-manipulation-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tictactoe-manipulation-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tictactoe-manipulation-eval

---


# tictactoe-manipulation-eval

> DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation — Wan et al. (2020) (arXiv:2005.02588, 2020)

## What this evaluates

Evaluates a robot's ability to perform adversarial interaction and temporal reasoning through a structured pick-and-place board game. It tests the integration of vision-based contour segmentation, game-state reasoning via Minimax, and precise motion planning under hardware constraints.

## Datasets

- **Yale-CMU-Berkeley objects set** — total ?; splits: (unstated)

## Metrics

- `t_sub` **(primary)** — range: seconds
  - Time in seconds to complete one pick-and-place sub-task, explicitly excluding gripper closing time. The metric is averaged over 10 repeated runs per robot cell.

## Input / output format

**Input**: RGB image of the 3x3 checkerboard and colored cubes; game state for Minimax planning.

**Output**: Pick and place actions for alternating turns until win or tie.

## Scoring recipe

```python
def score_tictactoe(predictions, gold):
    cycle_times = []
    for action in predictions:
        t = measure_time(action) - gripper_close_time
        cycle_times.append(t)
    return sum(cycle_times) / len(cycle_times)
```

## Common pitfalls

- Gripper closing time varies by hardware and must be explicitly subtracted from the reported time metric.
- The paper mistakenly refers to 'bin-clearing task' in the Tic-Tac-Toe section; ensure context matches the board game protocol.

## Evidence (verbatim from paper)

> The time to complete a pick and place sub-task $t_{sub}$ is recorded as a performance metric. To compare the performance of the robot arm, we purposely exclude the gripper closing time from $t_{sub}$ . We performed the bin-clearing task ten times on each robot cell, and the performance metric is averaged over ten repeated tasks.

## Citation

```bibtex
@misc{wan2020deepclaw,
  title={DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation},
  author={Wan et al. (2020)},
  year={2020},
  note={arXiv:2005.02588}
}
```

- arXiv: 2005.02588