# Clevr3d Vqa Eval

> Evaluates 3D visual question answering capabilities on point cloud scenes, probing spatial reasoning, object recognition, and scene graph understanding without relying on common-sense spatial priors. Use when the user wants to benchmark on CLEVR3D, or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/clevr3d-vqa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/clevr3d-vqa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/clevr3d-vqa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/clevr3d-vqa-eval

---


# clevr3d-vqa-eval

> Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation — Yan et al. (2021) (arXiv:2112.11691, 2021)

## What this evaluates

Evaluates 3D visual question answering capabilities on point cloud scenes, probing spatial reasoning, object recognition, and scene graph understanding without relying on common-sense spatial priors.

## Datasets

- **CLEVR3D** — total 60105; splits: train (49650), test (10455); repo https://github.com/yanx27/CLEVR3D

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Overall Accuracy = (number of correctly predicted answers) / (total number of test questions). Class Average Accuracy is the mean of per-class accuracies across different question types.
- `Top-k Recall` — range: [0, 1]
  - Fraction of ground truth objects, predicates, or relationship triplets that appear within the top-k predicted ranks.

## Input / output format

**Input**: 3D point cloud scene (objects sampled to 4,000 points each, augmented with random z-axis rotation) paired with a natural language question processed via a pre-trained BERT encoder.

**Output**: A single predicted answer class from the predefined VQA vocabulary, output by a classifier layer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_topk_recall(predictions, gold_labels, k):
    hits = 0
    for pred_rank, gold in zip(predictions, gold_labels):
        if gold in pred_rank[:k]:
            hits += 1
    return hits / len(gold_labels)
```

## Common pitfalls

- Models may exploit 2D spatial priors (e.g., tables always near sofas) if not trained on compositional scene manipulation data.
- Using bird's-eye view (BEV) 2D projections can cause occlusion and loss of true 3D spatial relationships, degrading performance.
- Accuracy can be inflated by class imbalance; reporting Class Average Accuracy across question types is necessary for fair comparison.

## Evidence (verbatim from paper)

> We adopt Accuracy as the evaluation metric and present the results of different question types on the CLEVR3D-REAL dataset are shown in Table III. ... we adopt the Top-k Recall score as the metric. ... We evaluate the model on 160 object and 26 predicate classes with [[10]] in Table V. ... separately evaluates the predicate (relationship) prediction in isolation from the object classes, where we adopt the Top-k Recall score as the metric.

## Citation

```bibtex
@misc{yan2021clevr3d,
  title={Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation},
  author={Yan et al. (2021)},
  year={2021},
  note={arXiv:2112.11691}
}
```

- arXiv: 2112.11691