Cameramotion Vqa Eval

cameramotion-vqa-eval

Geometry-Guided Camera Motion Understanding in VideoLLMs — Feng et al. (2026) (arXiv:2603.13119, 2026)

What this evaluates

Evaluates fine-grained camera motion recognition in VideoLLMs using multiple-choice questions and multi-label classification. Probes whether models can distinguish geometric camera movements from object motion and static shots.

Datasets

• CameraMotionVQA — total ?; splits: test (-1)
• CameraMotionDataset — total ?; splits: test (-1)

Metrics

• answer accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted options in the 4-way multiple-choice setting.
• instance accuracy — range: [0, 1]
  • Exact match accuracy: 1 if all predicted labels match the ground truth labels for a sample, else 0.
• Macro-F1 — range: [0, 1]
  • Unweighted mean of F1 scores computed independently for each motion primitive class.
• Weighted-F1 — range: [0, 1]
  • F1 score averaged across classes, weighted by the number of true instances for each class.

Input / output format

Input: Video segment/clips paired with a 4-way multiple-choice question prompt (VQA) or raw video frames for multi-label classification.

Output: Single selected option (VQA) or a multi-label vector over predefined camera motion primitives (classification).

Scoring recipe

def compute_metrics(preds, golds):
    # VQA / Instance Accuracy
    correct = sum(1 for p, g in zip(preds, golds) if p == g)
    acc = correct / len(golds)
    
    # Multi-label Instance Accuracy
    inst_acc = sum(1 for p, g in zip(preds, golds) if np.array_equal(p, g)) / len(golds)
    
    # Macro & Weighted F1
    macro_f1 = f1_score(golds, preds, average='macro', zero_division=0)
    weighted_f1 = f1_score(golds, preds, average='weighted', zero_division=0)
    return acc, inst_acc, macro_f1, weighted_f1

Common pitfalls

• Models frequently confuse geometrically similar primitives (e.g., truck vs. pan).
• Salient object motion often causes inconsistent directional predictions (e.g., pan-left vs. pan-right).
• Static segments are out-of-distribution for 3D foundation models, leading to unreliable predictions.

Evidence (verbatim from paper)

CameraMotionVQA formats each segment as a 4-way multiple-choice question, and models are evaluated using answer accuracy. For evaluating the camera motion multi-label task, we report instance accuracy (exact match of all labels), Macro-F1 across motion primitives, and Weighted-F1 weighted by label frequency.

Citation

@misc{feng2026geometryguided,
  title={Geometry-Guided Camera Motion Understanding in VideoLLMs},
  author={Feng et al. (2026)},
  year={2026},
  note={arXiv:2603.13119}
}

• arXiv: 2603.13119