Physically Disentangled Eval

physically-disentangled-eval

Physically Disentangled Representations — Klinghoffer et al. (2022) (arXiv:2204.05281, 2022)

What this evaluates

Evaluates the utility of physically disentangled scene representations (geometry, albedo, lighting, camera) for downstream vision tasks. Probes robustness to out-of-distribution lighting and viewpoints, and measures how well learned features transfer to clustering, linear classification, and segmentation benchmarks.

Datasets

• CelebA — total 6000; splits: test (6000)
• Buffy — total 568; splits: test (568)
• BBT — total 644; splits: test (644)
• RAF-DB — total 3068; splits: test (3068)
• CelebA Mask — total 27012; splits: train (24127), test (2885)
• ShapeNet Cars — total 1000; splits: test (1000)

Metrics

• clustering_accuracy (primary) — range: [0, 1]
  • Weighted cluster purity: Acc = (1/N) * sum_{c=1}^{|C|} n_c * p_c, where N is total samples, n_c is samples in cluster c, and p_c is the fraction of the most frequent class in cluster c relative to n_c.
• F1-score — range: [0, 1]
  • Weighted average F1-score computed over clusters or classes.
• mIoU — range: [0, 1]
  • Mean intersection over union across all segmentation classes.
• pixel_accuracy — range: [0, 1]
  • Fraction of correctly classified pixels across the entire dataset.

Input / output format

Input: 64x64 RGB images (or video frames). For downstream tasks, the model extracts 1x256 geometry and albedo feature embeddings per image.

Output: Cluster assignments per sample, predicted class labels, or pixel-wise segmentation masks.

Scoring recipe

def clustering_accuracy(labels, clusters):
    N = len(labels)
    acc = 0.0
    for c in unique_clusters:
        n_c = count(labels in c)
        p_c = max(count(label in c for label in unique_labels)) / n_c
        acc += n_c * p_c
    return acc / N

def linear_accuracy(predictions, gold):
    return sum(p == g for p, g in zip(predictions, gold)) / len(gold)

def miou(pred_masks, gt_masks, num_classes):
    ious = []
    for cls in range(num_classes):
        intersection = (pred_masks == cls) & (gt_masks == cls)
        union = (pred_masks == cls) | (gt_masks == cls)
        ious.append(intersection.sum() / union.sum())
    return sum(ious) / num_classes

Common pitfalls

• For video datasets (Buffy/BBT), features must be averaged per video track before clustering, not evaluated frame-by-frame.
• Linear classification experiments require varying the labeled training set size (100, 500, or 1000 samples) and testing both frozen and fine-tuned pre-trained encoders.
• Only geometry and albedo features are used for downstream tasks unless explicitly stated otherwise; other physical parameters are ignored.

Evidence (verbatim from paper)

To measure clustering performance, we report clustering accuracy (also known as weighted cluster purity) and F1-score. F1-score is computed using a weighted average. Clustering accuracy is computed by assigning the most common ground truth label for a cluster to all points in the cluster, as defined below. ... For linear classification, we report accuracy using a threshold of 50%. For segmentation, we report mean intersection over union (mIoU) and pixel accuracy.

Citation

@misc{klinghoffer2022physicallydisentangled,
  title={Physically Disentangled Representations},
  author={Klinghoffer et al. (2022)},
  year={2022},
  note={arXiv:2204.05281}
}

• arXiv: 2204.05281