Pose Aware Ssl Eval

pose-aware-ssl-eval

Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization — Wang et al. (2024) (arXiv:2403.14973, 2024)

What this evaluates

Evaluates the ability of self-supervised visual representations to capture geometric pose information and semantic content. It probes absolute and relative pose estimation accuracy, as well as semantic classification performance, across in-domain, out-of-domain, and real-world settings.

Datasets

• Carvana — total 5088; splits: test (5088)
• Synthetic dataset [8] — total ?; splits: train (-1)

Metrics

• semantic classification accuracy — range: percent
  • Accuracy of a linear probe trained on frozen 512-dim feature representations.
• absolute pose estimation accuracy — range: percent
  • Accuracy of a weighted k-nearest neighbor classifier on frozen feature representations.
• relative pose estimation accuracy (primary) — range: percent
  • Accuracy of a two-layer MLP probe on concatenated 1024-dim features from two views, predicting discretized relative pose deltas.

Input / output format

Input: RGB images of objects from varying viewpoints. Single images for absolute pose and semantic classification. Pairs of images for relative pose estimation.

Output: Discretized semantic class labels. Discretized absolute pose coordinates/angles. Discretized relative pose deltas.

Scoring recipe

def evaluate(features, labels, task):
    if task == 'semantic':
        logits = linear_probe(features)
        return accuracy(logits, labels)
    elif task == 'absolute_pose':
        preds = weighted_knn_predict(features, gallery_features, gallery_labels, k=5)
        return accuracy(preds, labels)
    elif task == 'relative_pose':
        z1, z2 = encoder(img1), encoder(img2)
        concat_feat = concatenate(z1, z2)
        preds = mlp_probe(concat_feat)
        return accuracy(preds, relative_labels)

Common pitfalls

• Using the last feature layer instead of mid-layer representations (e.g., 'conv3') significantly underestimates pose estimation performance, missing up to 20% gains.
• Evaluating only on in-domain synthetic data ignores the critical out-of-domain generalization capability that the method aims to improve.

Evidence (verbatim from paper)

We evaluate with a linear classification on top of the frozen representation from the feature layer with dimension 512. We employ a weighted k-nearest neighbor classifier as used in [56] on the representations from the feature layer. These representations are concatenated (resulting in a 1024-dim feature), and a simple probe of a two-layer perceptron with a hidden dimension of 1024 is used to predict the relative pose. Relative pose estimation is more computationally efficient but is generally harder as it relies on only two views for inference.

Citation

@misc{wang2024poseaware,
  title={Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2403.14973}
}

• arXiv: 2403.14973