Videgothink Eval

videgothink-eval

VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI — Cheng et al. (2024) (arXiv:2410.11623, 2024)

What this evaluates

Egocentric video understanding for embodied AI, probing capabilities in video question-answering, hierarchical task planning, visual grounding, and reward modeling. It evaluates how well multimodal models comprehend first-person, action-oriented video contexts required for robotic interaction.

Datasets

• VidEgoThink — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers, labels, or grounded regions compared to ground truth annotations. Computed per sub-task and averaged across object, action, and scene dimensions for QA, and across planning, grounding, and reward modeling tasks.

Input / output format

Input: Egocentric video clips (or sampled keyframes/captions for API models) paired with task-specific questions or instructions.

Output: Natural language responses (for QA, planning, feedback), bounding boxes or temporal segments (for visual grounding), or binary classifications (for critique).

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if normalize_text(pred) == gold:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• GPT-4o's privacy filters trigger refusals on indoor egocentric videos, causing performance drops when more frames are provided rather than improving them.
• Open-source MLLMs frequently fail to follow strict output formats for hierarchy planning, leading to artificially low accuracy scores due to format mismatches.
• Visual grounding tasks use novel formats not optimized for existing MLLMs, making direct performance comparisons with specialized grounding models less straightforward.

Evidence (verbatim from paper)

MLLMs perform poorly, with a best average accuracy of 32.82% across all dimensions (35.00% for object, 28.33% for action, and 26.33% for scene elements), indicating struggles with egocentric video question-answering.

Citation

@misc{cheng2024videgothink,
  title={VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2410.11623}
}

• arXiv: 2410.11623