Exact Eval

exact-eval

ExAct: A Video-Language Benchmark for Expert Action Analysis — Yi et al. (2025) (arXiv:2506.06277, 2025)

What this evaluates

Evaluates video-language models' ability to answer questions about expert-level physical skilled activities in long-form videos. It probes fine-grained action recognition, temporal reasoning, and domain-specific generalization across sports, bike repair, cooking, health, music, and dance.

Datasets

• ExAct — total 3521; splits: test (3521)

Metrics

• accuracy (primary) — range: percent
  • Percentage of questions for which the model selects the correct answer. Computed per activity and averaged across all 11 activities.

Input / output format

Input: Video clips (uniformly sampled to 32 frames at 796x448 resolution) paired with a multiple-choice question about expert physical skilled activities.

Output: A selected answer choice from the provided options.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Frame extraction must use uniform sampling of exactly 32 frames at 796x448 resolution for all models to maintain fairness.
• Accuracy is aggregated by first computing per-activity/domain accuracy and then averaging across the 11 activities, rather than calculating a single global accuracy over all 3,521 samples.
• Models resize the 796x448 frames internally according to their own input requirements, which can introduce variability if not carefully controlled.

Evidence (verbatim from paper)

We use standard question-answering accuracy as the primary evaluation metric. Our benchmark includes a total of 3,521 QA samples spanning 11 fine-grained physical skilled activities. For each activity, we compute the accuracy as the percentage of questions for which the model selects the correct answer. To summarize performance across the dataset, we report the average accuracy across all 11 activities.

Citation

@misc{yi2025exact,
  title={ExAct: A Video-Language Benchmark for Expert Action Analysis},
  author={Yi et al. (2025)},
  year={2025},
  note={arXiv:2506.06277}
}

• arXiv: 2506.06277