Handy Vqa Eval

handy-vqa-eval

HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics — Tateno et al. (2025) (arXiv:2512.00885, 2025)

What this evaluates

Evaluates video foundation models' ability to understand fine-grained spatiotemporal dynamics in hand-object interactions. It probes spatial reasoning, motion tracking, and part-level geometric grounding through multiple-choice questions and video object segmentation tasks.

Datasets

• HanDyVQA — total 11100; splits: test (-1)

Metrics

• top-1 accuracy (primary) — range: [0, 1]
  • Fraction of questions where the model's predicted option matches the ground-truth answer. Averaged across Action, Process, Location, State, and Parts categories.
• Average Precision (AP) — range: [0, 1]
  • Area under the precision-recall curve for the Objects category, used because questions can have multiple correct answers.
• Jaccard Index (J) — range: [0, 1]
  • Intersection over union between predicted and ground-truth segmentation masks, averaged over all annotated frames in a video.
• Boundary F-measure (F) — range: [0, 1]
  • F-score computed between the boundaries of predicted and ground-truth masks, averaged over all annotated frames.

Input / output format

Input: Video clips (uniformly sampled to 16 frames) paired with a natural language question and a list of multiple-choice options. For ReasoningVOS, the input is the video plus either the question or the ground-truth option as a text prompt.

Output: For MCQ: a single selected option (letter or text). For ReasoningVOS: per-frame binary/instance segmentation masks corresponding to the queried hand, object, or object part.

Scoring recipe

# MCQ Scoring
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
accuracy = correct / len(golds)

# Objects AP Scoring
ap = compute_average_precision(gold_masks, pred_masks)

# ReasoningVOS Scoring
j_scores = []
f_scores = []
for frame in frames:
    j_scores.append(jaccard_index(pred_mask[frame], gold_mask[frame]))
    f_scores.append(boundary_f_measure(pred_mask[frame], gold_mask[frame]))
j_acc = mean(j_scores)
f_acc = mean(f_scores)

Common pitfalls

• Models often rely on superficial textual or contextual cues rather than actual video dynamics, leading to inflated scores on text-only baselines if not carefully controlled.
• The Objects category uses Average Precision instead of accuracy, which can cause confusion when computing the overall average score across categories.
• Error rates are computed per error type based on distractor analysis, but a single wrong answer may correspond to multiple error types, complicating aggregate error reporting.

Evidence (verbatim from paper)

Evaluation metrics. We report top-1 accuracy for all the categories except Objects, and Average Precision (AP) for Objects because it has more than one answers per question. Following standard VOS evaluation protocols [30, 49], we use the Jaccard Index (I) and Boundary F-measure (F) computed for each frame and report their average over annotated frames.

Citation

@misc{tateno2025handyvqa,
  title={HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics},
  author={Tateno et al. (2025)},
  year={2025},
  note={arXiv:2512.00885}
}

• arXiv: 2512.00885