Tuna Eval

tuna-eval

TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos — Kong et al. (2025) (arXiv:2505.20124, 2025)

What this evaluates

Evaluates fine-grained temporal understanding on dense dynamic videos, probing camera motion, scene transitions, action sequences, and multi-subject interactions. It measures how well models capture dynamic visual elements and handle varying video complexities.

Datasets

• TUNA — total 1000; splits: test (1000)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall computed over token-level matches between generated captions and ground truth.
• Accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered multiple-choice questions.

Input / output format

Input: Videos (default: 32 uniformly sampled frames) accompanied by prompts for open-ended captioning or multiple-choice questions.

Output: Free-form text captions or selected multiple-choice options.

Scoring recipe

def compute_f1(predictions, golds):
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        pred_tokens = set(pred.lower().split())
        gold_tokens = set(gold.lower().split())
        intersection = pred_tokens & gold_tokens
        prec = len(intersection) / len(pred_tokens) if pred_tokens else 0
        rec = len(intersection) / len(gold_tokens) if gold_tokens else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)

Common pitfalls

• Frame sampling strategy significantly impacts results; increasing frames beyond 32 can degrade performance on highly complex videos.
• Models often treat sampled frames as independent static images, failing to capture temporal dynamics and action sequences.
• Evaluation splits videos by complexity (dynamic level, number of subjects/scenes), requiring stratified reporting to avoid masking weaknesses.

Evidence (verbatim from paper)

Precision reflects the correctness of the content mentioned in the descriptions, while recall reflects the completeness of the descriptions. As shown in Table [2], majority of video LMMs achieve a precision over 70%, but recall is below 50%, indicating that many visual elements in videos are often overlooked or misdescribed. The state-of-the-art model GPT-4o only achieve an F1 score of 58.5%, with a recall of 48.2%, highlighting that LMMs still have a great potential for improvement in the task of temporally dense captioning.

Citation

@misc{kong2025tuna,
  title={TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos},
  author={Kong et al. (2025)},
  year={2025},
  note={arXiv:2505.20124}
}

• arXiv: 2505.20124