Clearpose Eval

clearpose-eval

ClearPose: Large-scale Transparent Object Dataset and Benchmark — Chen et al. (2022) (arXiv:2203.03890, 2022)

What this evaluates

Evaluates perception models' ability to estimate 6 DoF poses and complete depth maps for transparent and translucent objects. It specifically probes robustness to challenging real-world conditions such as heavy occlusion, cluttered backgrounds, varying lighting, and objects filled with liquid.

Datasets

• ClearPose — total 354481; splits: train (-1), test (-1); repo https://github.com/opipari/ClearPose

Metrics

• 6 DoF poses (primary) — range: other
  • Not explicitly defined in the provided text; typically measures translation distance and rotation angle between predicted and ground truth object poses.

Input / output format

Input: RGB-D frames (640x480) capturing transparent objects in various scenes.

Output: 6-DoF object poses, segmentation masks, surface normals, and fixed ground truth depth images.

Scoring recipe

def compute_metrics(pred_poses, gt_poses, pred_depth, gt_depth):
    t_err = np.linalg.norm(pred_poses[:, :3, 3] - gt_poses[:, :3, 3])
    r_err = np.arccos(np.clip((np.trace(pred_poses[:, :3, :3].T @ gt_poses[:, :3, :3]) - 1) / 2, -1, 1))
    depth_rmse = np.sqrt(np.mean((pred_depth - gt_depth) ** 2))
    return {'pose_error': t_err + r_err, 'depth_rmse': depth_rmse}

Common pitfalls

• Transparent objects yield broken or inconsistent depth measurements, requiring specialized handling beyond standard RGB-D pipelines.
• The test set is divided into 6 specific challenging scenarios (e.g., heavy occlusion, liquid-filled objects, non-planar surfaces) that are strictly held out from training to evaluate generalization.

Evidence (verbatim from paper)

In total there are 354,481 RGB-D frames captured in 51 scenes, with 5,052,429 object instance annotations with 6 DoF poses, segmentation masks, surface normals, and ground truth depth images. For the testing set, as the appearance of transparent objects depends on their context within a scene, we consider 6 different test cases and collect corresponding scenes as follows: (1) different backgrounds: novel backgrounds that never appeared in the training scenes with each object subset. (2) heavy occlusions: cluttered scenes each with about 25 objects that form multiple layers of occlusion when viewed from the table’s side. (3) translucent/transparent covers: scenes with all transparent objects placed inside a translucent box. (4) together with opaque objects: transparent objects placed together with opaque YCB and HOPE objects, which did not appear in the training set. (5) filled with liquid: scenes with transparent objects filled with different colored liquid. (6) non-planar configuration: scenes with objects placed onto different surfaces with multiple heights.

Citation

@misc{chen2022clearpose,
  title={ClearPose: Large-scale Transparent Object Dataset and Benchmark},
  author={Chen et al. (2022)},
  year={2022},
  note={arXiv:2203.03890}
}

• arXiv: 2203.03890