Clevr3d Vqa Eval

clevr3d-vqa-eval

Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation — Yan et al. (2021) (arXiv:2112.11691, 2021)

What this evaluates

Evaluates 3D visual question answering capabilities on point cloud scenes, probing spatial reasoning, object recognition, and scene graph understanding without relying on common-sense spatial priors.

Datasets

• CLEVR3D — total 60105; splits: train (49650), test (10455); repo https://github.com/yanx27/CLEVR3D

Metrics

• Accuracy (primary) — range: [0, 1]
  • Overall Accuracy = (number of correctly predicted answers) / (total number of test questions). Class Average Accuracy is the mean of per-class accuracies across different question types.
• Top-k Recall — range: [0, 1]
  • Fraction of ground truth objects, predicates, or relationship triplets that appear within the top-k predicted ranks.

Input / output format

Input: 3D point cloud scene (objects sampled to 4,000 points each, augmented with random z-axis rotation) paired with a natural language question processed via a pre-trained BERT encoder.

Output: A single predicted answer class from the predefined VQA vocabulary, output by a classifier layer.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_topk_recall(predictions, gold_labels, k):
    hits = 0
    for pred_rank, gold in zip(predictions, gold_labels):
        if gold in pred_rank[:k]:
            hits += 1
    return hits / len(gold_labels)

Common pitfalls

• Models may exploit 2D spatial priors (e.g., tables always near sofas) if not trained on compositional scene manipulation data.
• Using bird's-eye view (BEV) 2D projections can cause occlusion and loss of true 3D spatial relationships, degrading performance.
• Accuracy can be inflated by class imbalance; reporting Class Average Accuracy across question types is necessary for fair comparison.

Evidence (verbatim from paper)

We adopt Accuracy as the evaluation metric and present the results of different question types on the CLEVR3D-REAL dataset are shown in Table III. ... we adopt the Top-k Recall score as the metric. ... We evaluate the model on 160 object and 26 predicate classes with [[10]] in Table V. ... separately evaluates the predicate (relationship) prediction in isolation from the object classes, where we adopt the Top-k Recall score as the metric.

Citation

@misc{yan2021clevr3d,
  title={Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation},
  author={Yan et al. (2021)},
  year={2021},
  note={arXiv:2112.11691}
}

• arXiv: 2112.11691