Vector Eval

vector-eval

What Happens When: Learning Temporal Orders of Events in Videos — Ahn et al. (2025) (arXiv:2512.08979, 2025)

What this evaluates

Evaluates a model's ability to understand and reason about the temporal order of multiple events in long-form videos. It probes whether the model can correctly sequence events, identify relative ordering, and detect pattern anomalies across varying sequence lengths and difficulties.

Datasets

• VECTOR — total ?; splits: test (-1)

Metrics

• EM (Exact Match) (primary) — range: percent
  • Exact match accuracy: 1 if the predicted sequence of events exactly matches the ground-truth sequence, 0 otherwise. Averaged over all instances.

Input / output format

Input: A video represented by 32 sampled frames, accompanied by a text prompt asking for the temporal order of events or pattern anomaly detection.

Output: A predicted ordered list of events or a textual answer specifying the sequence.

Scoring recipe

def compute_em(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

Common pitfalls

• Simply increasing the number of input frames beyond a threshold does not consistently improve performance and can even degrade it.
• Open-source VLMMs drop to chance-level accuracy on longer sequences (L2, 8 events), highlighting struggles with long-range temporal reasoning.
• Order-agnostic metrics (OM) remain stable across conditions, so relying solely on them masks failures in true temporal-order comprehension.

Evidence (verbatim from paper)

Unless specified, we use 32 input frames per video. Specifically, Table 2 reports EM scores for Tasks1-3, while Table 3 provides additional sequencing metrics. All models show notable performance declines from L1 to L2; for example, GPT-4o and Gemini drop from approximately 75–83% EM at L1 to 50–60% at L2.

Citation

@misc{ahn2025what,
  title={What Happens When: Learning Temporal Orders of Events in Videos},
  author={Ahn et al. (2025)},
  year={2025},
  note={arXiv:2512.08979}
}

• arXiv: 2512.08979