Triplesumm Video Summarization Eval

triplesumm-video-summarization-eval

TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization — Sumin Kim et al. (2026) (arXiv:2603.01169, 2026)

What this evaluates

Evaluates a model's ability to perform video summarization by predicting frame-level importance scores across visual, textual, and audio modalities. It probes the model's capacity for adaptive multimodal fusion and temporal dependency modeling to identify salient segments in long videos.

Datasets

• MoSu — total ?; splits: original (-1); repo https://github.com/smkim37/TripleSumm
• Mr. HiSum — total ?; splits: original (-1)
• SumMe — total ?; splits: TV (-1), TVT (-1)
• TVSum — total ?; splits: TV (-1), TVT (-1)

Metrics

• Kendall’s τ (kTau) (primary) — range: [-1, 1]
  • Rank-based correlation coefficient measuring the ordinal association between predicted frame-level importance scores and ground-truth importance labels.
• Spearman’s ρ (sRho) (primary) — range: [-1, 1]
  • Rank-based correlation coefficient assessing how well the predicted importance scores preserve the rank order of ground-truth importance.
• mAP50 — range: [0, 1]
  • Mean Average Precision computed over the top 50% of 5-second segments ranked by predicted importance, evaluated against ground-truth highlights.
• mAP15 — range: [0, 1]
  • Mean Average Precision computed over the top 15% of 5-second segments ranked by predicted importance, evaluated against ground-truth highlights.

Input / output format

Input: Temporally aligned feature sequences of equal length for visual (CLIP frame embeddings at 1 fps), text (RoBERTa [CLS] tokens broadcast to frames), and audio (AST features at 1-sec intervals). For external datasets, official visual features plus generated captions and raw audio are used.

Output: Frame-level importance scores, which are subsequently ranked to identify salient segments.

Scoring recipe

def compute_metrics(pred_scores, gold_scores):
    # Rank-based correlations
    ktau = kendalltau(pred_scores, gold_scores)
    srho = spearmanr(pred_scores, gold_scores)
    # Segment-level highlight detection
    segments = split_video_into_segments(pred_scores, duration=5.0)
    top_50_idx = np.argsort(segments)[-int(len(segments)*0.5):]
    top_15_idx = np.argsort(segments)[-int(len(segments)*0.15):]
    mAP50 = compute_ap(top_50_idx, gold_scores)
    mAP15 = compute_ap(top_15_idx, gold_scores)
    return ktau, srho, mAP50, mAP15

Common pitfalls

• SumMe and TVSum evaluations historically suffer from overfitting when using only train/validation splits; the paper explicitly compares against both TV and TVT protocols to correct this.
• mAP metrics are computed on 5-second segments, not raw frames, which changes the granularity of highlight detection compared to frame-level importance.
• The model must handle missing modalities gracefully; performance drops significantly if modality weighting isn't adaptive to content changes.

Evidence (verbatim from paper)

Following RethinkingSumm, we primarily report rank-based correlation metrics, Kendall’s τ (kTau) and Spearman’s ρ (sRho), on the frame-level importance prediction, being consistent with recent studies (CSTA; MAAM; SummDiff). On MoSu and Mr. HiSum datasets where the Most Replayed statistics are used as ground-truth, we further assess the highlight detection performance with mean Average Precision (mAP) following Mr.HiSum. Following this protocol, each video is divided into 5-second segments and ranked by predicted importance. The top 50% (mAP50) and 15% (mAP15) of these segments are then evaluated as the predicted highlights against the ground-truth.

Citation

@misc{kim2026triplesumm,
  title={TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization},
  author={Sumin Kim et al. (2026)},
  year={2026},
  note={arXiv:2603.01169}
}

• arXiv: 2603.01169