Egoschema Eval

egoschema-eval

EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding — Mangalam et al. (2023) (arXiv:2308.09126, 2023)

What this evaluates

Probes long-term visual memory and temporal reasoning in video-language models by requiring them to answer multiple-choice questions about very long-form videos. It measures the model's ability to retain and retrieve information across extended durations without relying on short clip analysis.

Datasets

• EgoSchema — total ?; splits: test (-1); repo https://github.com/egoschema/EgoSchema

Metrics

• QA Accuracy (primary) — range: percent
  • Percentage of correctly predicted answer options out of the total number of questions. Calculated as (number of correct predictions / total questions) × 100.

Input / output format

Input: Video clip with uniformly sampled frames (e.g., 1 to 90 frames) + multiple-choice question text + answer options.

Output: Predicted answer option (A, B, C, or D) or probability scores per option to select the highest.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Frame sampling strategy affects performance non-monotonically; increasing frames does not always improve accuracy.
• GPU memory constraints limit the maximum number of frames that can be processed, creating inconsistent evaluation settings across models.
• Human baseline accuracy varies significantly based on time pressure and modality constraints (e.g., video-only vs. video+text).

Evidence (verbatim from paper)

In Table 6, We benchmark several state-of-the-art video-language models, with the intention of adding more models in the future, in a Zero-shot question-answering setting on EgoSchema. We evaluate each model in at least two settings. First is the conventional inference setting, where the model is assessed based on the same number of frames it was trained with. And second is a less challenging setting, where the model is tested on the maximum number of frames possible to execute inference with, using an 80G A100, without exceeding the GPU memory capacity. In both settings, frames are sampled uniformly from the input video clip. We observe accuracy to be non-monotonic in frame length, and report results in 1 to 30 frames in Table 6.

Citation

@misc{mangalam2023egoschema,
  title={EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding},
  author={Mangalam et al. (2023)},
  year={2023},
  note={arXiv:2308.09126}
}

• arXiv: 2308.09126