Precision Recall F1 T

precision-recall-f1-t

SoccerHigh: A Benchmark Dataset for Automatic Soccer Video Summarization — Díaz-Juan et al. (2025) (arXiv:2509.01439, 2025)

What this evaluates

Evaluates a model's ability to identify and rank key moments (shots) in soccer match videos for summarization. It measures how well the model selects representative content when constrained to match the exact duration of a human-curated highlight summary.

Datasets

• SoccerHigh — total 237; splits: test (-1)

Metrics

• Precision@$T$ — range: [0, 1]
  • Standard precision computed on the set of selected shots after constraining the predicted summary duration to exactly match the ground truth duration $T$.
• Recall@$T$ — range: [0, 1]
  • Standard recall computed on the set of selected shots after constraining the predicted summary duration to exactly match the ground truth duration $T$.
• F1 Score@$T$ (primary) — range: [0, 1]
  • Harmonic mean of Precision@$T$ and Recall@$T$, computed after aligning the predicted summary length to the ground truth duration $T$.

Input / output format

Input: Broadcast soccer video with annotated shots and importance scores; ground truth highlight summary with total duration $T$.

Output: Ranked list of selected shots forming a summary whose cumulative duration exactly matches the ground truth duration $T$.

Scoring recipe

def compute_metrics_t(predictions, ground_truth, T):
    selected = []
    current_dur = 0
    for shot in sorted(predictions, key=lambda x: x['score'], reverse=True):
        selected.append(shot['id'])
        current_dur += shot['duration']
        if current_dur >= T:
            break
    gt_set = set(gt['id'] for gt in ground_truth)
    pred_set = set(selected)
    tp = len(pred_set & gt_set)
    precision = tp / len(pred_set) if pred_set else 0
    recall = tp / len(gt_set) if gt_set else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1

Common pitfalls

• Assuming the model must output a fixed percentage (e.g., 15%) of the original video; the constraint only applies to the evaluation alignment, not the model's generation.
• Computing standard precision/recall without first truncating the predicted summary to match the ground truth duration $T$, which would penalize models for selecting valid but extra content.

Evidence (verbatim from paper)

We denote the temporal length of each ground truth summary as $T$ and define Precision$@T$, Recall$@T$, and the F1 Score$@T$ as evaluation metrics constrained by this length. During inference, predicted shots are ranked according to their importance scores, and only the top-ranked shots are selected until the cumulative duration matches the ground truth summary.

Citation

@misc{diazjuan2025soccerhigh,
  title={SoccerHigh: A Benchmark Dataset for Automatic Soccer Video Summarization},
  author={Díaz-Juan et al. (2025)},
  year={2025},
  note={arXiv:2509.01439}
}

• arXiv: 2509.01439