Octscenes Eval

octscenes-eval

OCTScenes: A Versatile Real-World Dataset of Tabletop Scenes for Object-Centric Learning — Huang et al. (2023) (arXiv:2306.09682, 2023)

What this evaluates

Evaluates object-centric learning models on real-world tabletop scenes to measure their ability to segment foreground objects and background, as well as reconstruct scenes from single-image, video, or multi-view inputs.

Datasets

• OCTScenes-A — total ?; splits: train (90000), val (3000), test (3000)
• OCTScenes-B — total ?; splits: train (144000), val (3000), test (3000)

Metrics

• ARI-O (primary) — range: [0, 1]
  • Adjusted Rand Index computed only on object pixels. Measures the similarity between predicted and ground-truth object clusters, adjusted for chance. Higher values indicate better segmentation.
• AMI-O — range: [0, 1]
  • Adjusted Mutual Information computed only on object pixels. Measures mutual information between predicted and ground-truth clusters, adjusted for chance. Higher values indicate better segmentation.
• mIoU — range: [0, 1]
  • mean Intersection over Union. Computes the average IoU across all object classes/predicted clusters. Higher values indicate better overlap with ground truth.
• MSE — range: [0, inf)
  • Mean Squared Error between reconstructed and original images at the pixel level. Lower values indicate better reconstruction, though it favors blurry results.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity. Measures perceptual difference in feature space between reconstructed and original images. Lower values indicate better reconstruction aligned with human perception.

Input / output format

Input: RGB-D images resized to 128x128. Scenes are split into sub-scenes with 10-frame intervals. Inputs vary by method type: single-frame images, video sequences, or multi-view frames.

Output: Predicted object segmentation masks (cluster assignments) and reconstructed RGB images.

Scoring recipe

def evaluate(pred_masks, gt_masks, pred_img, gt_img):
    ari_o = adjusted_rand_index(pred_masks, gt_masks, ignore_bg=True)
    ami_o = adjusted_mutual_info(pred_masks, gt_masks, ignore_bg=True)
    miou = mean_intersection_over_union(pred_masks, gt_masks)
    mse = np.mean((pred_img - gt_img) ** 2)
    lpips = perceptual_similarity(pred_img, gt_img)
    return {'ARI-O': ari_o, 'AMI-O': ami_o, 'mIoU': miou, 'MSE': mse, 'LPIPS': lpips}

Common pitfalls

• Models often fail to segment the background as a single cluster, instead splitting it into multiple parts to capture complex background variations.
• Reconstruction metrics can be inconsistent: pixel-level MSE may be low while feature-level LPIPS is high (or vice versa), depending on whether the model uses a mixture-based or transformer-based decoder.
• Methods relying on random slot initialization (e.g., SLATE, STEVE) exhibit high performance variance across different random seeds and hyperparameters.

Evidence (verbatim from paper)

We assess segmentation quality with Adjusted Rand Index (ARI) [21], Adjusted Mutual Information (AMI) [36], and mean Intersection over Union (mIoU). ARI and AMI, which measure the congruence between two data clusters, are robust indicators of superior segmentation performance as their values increase. mIoU, a standard metric for evaluating object segmentation, provides a quantifiable measure of the overlap between the predicted and ground truth segmentation. We further refine our analysis by introducing the terms AMI-A and ARI-A, signifying calculations that consider both the objects and the background, and AMI-O and ARI-O, which focus solely on the objects. We rely on Minimize Squared Error (MSE) and Learned Perceptual Image Patch Similarity (LPIPS) [37] to evaluate the quality of reconstruction, both of which indicate better reconstruction performance at lower values.

Citation

@misc{huang2023octscenes,
  title={OCTScenes: A Versatile Real-World Dataset of Tabletop Scenes for Object-Centric Learning},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2306.09682}
}

• arXiv: 2306.09682