Vista Eval

vista-eval

What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentations — Liu et al. (2025) (arXiv:2502.08279, 2025)

What this evaluates

This benchmark evaluates a model's ability to generate concise, structured summaries of scientific conference talks from video inputs. It specifically probes informativeness, alignment with visual/audio content, and factual consistency against the corresponding paper abstracts.

Datasets

• VISTA — total 18599; splits: test (-1)

Metrics

• ROUGE-1 F1 (primary) — range: [0, 1]
  • F1 score of unigram overlap between the generated summary and the reference abstract.
• ROUGE-2 F1 — range: [0, 1]
  • F1 score of bigram overlap between the generated summary and the reference abstract.
• ROUGE-LSum F1 — range: [0, 1]
  • Summary-level ROUGE F1 score that treats each paragraph as a separate document.
• SacreBLEU — range: [0, 100]
  • BLEU score computed with standardized tokenization and a fixed reference corpus.
• METEOR — range: [0, 1]
  • Alignment-based metric incorporating synonymy, stemming, and word order penalties.
• BERTScore — range: [0, 1]
  • Cosine similarity between contextual embeddings of generated and reference tokens, aggregated via F1.
• CIDEr-D — range: [0, 1]
  • TF-IDF weighted n-gram similarity score designed for image and video captioning evaluation.
• VideoScore — range: [0, 1]
  • Metric evaluating how well the generated summary aligns with the input video content.
• FactVC — range: [0, 1]
  • Metric measuring factual consistency between the generated summary and the source video/abstract.

Input / output format

Input: Video files (containing visual and audio modalities), optionally pre-processed into transcripts or OCR text, accompanied by standardized prompt instructions.

Output: A text summary of the scientific presentation, intended to align with the structure and content of the corresponding paper abstract.

Scoring recipe

def compute_metrics(predictions, references, videos):
    scores = {}
    scores['ROUGE-1 F1'] = rouge_score(references, predictions, 'rouge1').fmeasure
    scores['ROUGE-2 F1'] = rouge_score(references, predictions, 'rouge2').fmeasure
    scores['ROUGE-LSum F1'] = rouge_score(references, predictions, 'rougeLsum').fmeasure
    scores['SacreBLEU'] = sacrebleu.corpus_bleu(predictions, [references]).score
    scores['METEOR'] = meteor_score(references, predictions)
    scores['BERTScore'] = bertscore_score(references, predictions, lang='en').f1.mean()
    scores['CIDEr-D'] = cider_score(references, predictions)
    scores['VideoScore'] = videoscore_model.evaluate(predictions, videos)
    scores['FactVC'] = factvc_model.evaluate(predictions, videos, references)
    return scores

Common pitfalls

• Models may inadvertently rely solely on audio transcripts or OCR text rather than multimodal video features, skewing alignment scores if not explicitly controlled.
• Factual consistency is particularly challenging for technical scientific content where visual slides contain critical data not present in the audio or abstract.
• Zero-shot, QLoRA, and full fine-tuning settings produce vastly different baselines; comparing them without identical hyperparameter settings leads to unfair benchmarking.

Evidence (verbatim from paper)

We report a set of evaluation metrics to measure informativeness, alignment, and factual consistency in summaries. For informativeness, we utilize ROUGE, SacreBLEU, METEOR, BERTScore, and CIDEr-D. Specifically, we provide the F1 scores for Rouge-1 (R1), Rouge-2 (R2), and Rouge-LSum (RLSUM). Alignment to the input video is evaluated with VideoScore, and factual consistency with FactVC.

Citation

@misc{liu2025vista,
  title={What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentations},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2502.08279}
}

• arXiv: 2502.08279