Sgaligner Eval

sgaligner-eval

SGAligner : 3D Scene Alignment with Scene Graphs — Sayan Deb Sarkar et al. (2023) (arXiv:2304.14880, 2023)

What this evaluates

Evaluates the ability to align 3D scene graphs by matching semantic entities across scenes with varying spatial overlap and environmental changes. It further tests downstream 3D point cloud registration and mosaicking capabilities using the predicted node alignments to initialize geometric correspondence extraction.

Datasets

• 3RScan (generated sub-scene pairs) — total 1335; splits: train (15102), test (1932)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of 1/rank for each query node, where rank is the position of the correct match in the cosine similarity ranking.
• Hits@K — range: [0, 1]
  • Ratio of correct matches appearing within the top K ranked predictions based on cosine similarity.
• SGAR — range: [0, 1]
  • Scene Graph Alignment Recall: proportion of correctly aligned nodes when using top-K, top-50%, or all predicted matches to initialize alignment.
• Chamfer Distance (CD) — range: other
  • Mean squared distance between corresponding points in two registered point clouds.
• Relative Rotation Error (RRE) — range: degrees
  • Angular error in degrees between the predicted and ground truth rotation matrices.
• Relative Translation Error (RTE) — range: cm
  • Linear error in centimeters between the predicted and ground truth translation vectors.
• Registration Recall (RR) — range: percent
  • Percentage of scene pairs successfully registered, defined as RMSE < 0.2.

Input / output format

Input: Pairs of 3D scene graphs containing semantically annotated object instances (point clouds downsampled to 512 points via farthest point sampling), attributes, and relationships.

Output: Ranked list of matched entity nodes between the two scene graphs, used to initialize 3D point cloud registration.

Scoring recipe

def compute_node_alignment_metrics(predictions, gold):
    ranks = []
    hits = {k: 0 for k in [1,2,3,4,5]}
    for q_id, correct_id in gold.items():
        pred_list = predictions[q_id]
        if correct_id in pred_list:
            rank = pred_list.index(correct_id) + 1
            ranks.append(1.0 / rank)
            for k in hits:
                if rank <= k:
                    hits[k] += 1
    mrr = sum(ranks) / len(ranks)
    hits_at_k = {k: v / len(gold) for k, v in hits.items()}
    return mrr, hits_at_k

Common pitfalls

• Performance drops significantly when using predicted scene graphs instead of ground truth, especially under semantic noise or wrong labels.
• Higher K values for node matching (e.g., K=3) can degrade downstream registration performance by introducing more outliers for RANSAC, despite higher Hits@K scores.
• Overlap percentage heavily influences node matching accuracy, but the method remains robust down to 10-30% overlap.

Evidence (verbatim from paper)

We utilize cosine similarity on the joint embedding to calculate the similarity between two matched entities and employ Mean Reciprocal Rank (MRR) and Hits@K, where K = {1,2,...,5}. MRR denotes the mean reciprocal rank of correct matches. Hits@K denotes the ratio of correct matches appearing within top K, based on their cosine similarity ranking.

Citation

@misc{sarkar2023sgaligner,
  title={SGAligner : 3D Scene Alignment with Scene Graphs},
  author={Sayan Deb Sarkar et al. (2023)},
  year={2023},
  note={arXiv:2304.14880}
}

• arXiv: 2304.14880