Fiova Eval

fiova-eval

FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning — Hu et al. (2024) (arXiv:2410.15270, 2024)

What this evaluates

Evaluates the ability of Large Vision-Language Models (LVLMs) to generate accurate, fluent, and comprehensive captions for long videos. It probes semantic alignment, event coverage, and hallucination resistance by comparing model outputs against multi-annotator human references.

Datasets

• FIOVA — total 3002; splits: test (-1)

Metrics

• BLEU — range: [0, 1]
  • Standard n-gram overlap metric measuring exact lexical matches between prediction and ground truth.
• METEOR — range: [0, 1]
  • Metric that aligns predictions and references using stemming, synonymy, and exact matches, weighting precision and recall.
• GLEU — range: [0, 1]
  • Generative Language Understanding Evaluation; measures n-gram overlap with a penalty for repetition.
• FIOVA-DQ F1 (primary) — range: [0, 1]
  • Event-level metric that extracts events from the model caption and ground truth, then computes F1 based on event overlap weighted by cognitive weights derived from multi-annotator consensus.
• FIOVA-DQ Precision — range: [0, 1]
  • Proportion of events in the model caption that are correctly identified and weighted by annotator consensus in the ground truth.
• FIOVA-DQ Recall — range: [0, 1]
  • Proportion of consensus-weighted events in the ground truth that are successfully captured by the model caption.

Input / output format

Input: Raw video file (average duration 33.6 seconds)

Output: Natural language caption describing the video content

Scoring recipe

def score_video_caption(pred_caption, gt_caption):
    # Traditional metrics
    bleu = compute_bleu(pred_caption, gt_caption)
    meteor = compute_meteor(pred_caption, gt_caption)
    gleu = compute_gleu(pred_caption, gt_caption)
    
    # FIOVA-DQ event-level scoring
    events_pred = extract_events(pred_caption)
    events_gt = extract_events(gt_caption)  # aggregated from 5 annotators
    weights = compute_consensus_weights(events_gt)  # based on annotator agreement
    
    matched = count_weighted_matches(events_pred, events_gt, weights)
    precision = matched / max(len(events_pred), 1)
    recall = matched / max(len(events_gt), 1)
    f1 = 2 * (precision * recall) / (precision + recall + 1e-8)
    
    return {'BLEU': bleu, 'METEOR': meteor, 'GLEU': gleu, 
            'FIOVA-DQ_Precision': precision, 'FIOVA-DQ_Recall': recall, 'FIOVA-DQ_F1': f1}

Common pitfalls

• Using a single human annotation as ground truth instead of the aggregated multi-annotator reference, which artificially lowers metric scores and misrepresents model performance.
• Relying solely on traditional n-gram metrics (BLEU/METEOR) which show near-zero correlation with human preference and event-level quality.
• Ignoring the cognitive weighting scheme in FIOVA-DQ, which adjusts for inter-subjective agreement rather than treating all events equally.

Evidence (verbatim from paper)

We computed Spearman correlations between the human rankings and automatic rankings derived from six metrics: F1, Precision, and Recall under FIOVA-DQ, and F1, Precision, and Recall under AutoDQ. Tab. A5 presents detailed results for each evaluator.

Citation

@misc{hu2024fiova,
  title={FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning},
  author={Hu et al. (2024)},
  year={2024},
  note={arXiv:2410.15270}
}

• arXiv: 2410.15270