qhjqhj00 mobile-construction-eval
Simultaneous Navigation and Construction Benchmarking Environments — Han et al. (2021) (arXiv:2103.16732, 2021)
What this evaluates
Tests a robot's ability to simultaneously navigate and construct a target structure in 1D/2D/3D grid worlds under partial observability and environmental uncertainty. It evaluates the bi-directional coupling between localization and manipulation planning.
Datasets
- Mobile Construction Benchmark — total ?; splits: test (-1)
Metrics
IoU(primary) — range: [0, 1]- Intersection over Union between the terminal grid state G^T and the ground-truth design D. Computed as the sum over all grid cells l of min(G^T(l), D(l)) / max(G^T(l), D(l)).
Input / output format
Input: Partially observable grid world state with a limited sensing region; target design D (static) or dynamic curve/triangle sequence (dynamic).
Output: Action sequence consisting of 'move' or 'drop brick' commands to construct the target design.
Scoring recipe
def compute_iou(terminal_grid, target_design):
intersection = sum(min(terminal_grid[l], target_design[l]) for l in grid_cells)
union = sum(max(terminal_grid[l], target_design[l]) for l in grid_cells)
return intersection / union if union > 0 else 0.0
Common pitfalls
- IoU is evaluated only at the terminal state, ignoring intermediate construction progress or partial matches.
- Dynamic tasks use fixed groups of designs for testing, which may not reflect generalization to unseen dynamic patterns.
- 3D environments introduce obstacles and agent/obstacle volume, drastically increasing difficulty and often causing agents to learn to only move rather than build.
Evidence (verbatim from paper)
Evaluation metric: we use the IoU score as our evaluation criteria which is measured between the terminal grid state $G^{T}$ and the ground-truth design $D$. Here, we use IoU as our evaluation criteria because it is more straightforward for us to evaluate the quality of the built structure to the ground-truth design $D$. The IoU is defined as $IoU=\frac{G^{T}\cap D}{G^{T}\cup D}=\sum_{l\in\mathcal{L}}\frac{\min(G^{T}(l),D(l))}{\max(G^{T}(l),D(l))}.$
Citation
@misc{han2021simultaneous,
title={Simultaneous Navigation and Construction Benchmarking Environments},
author={Han et al. (2021)},
year={2021},
note={arXiv:2103.16732}
}
- arXiv: 2103.16732