Video Prediction Eval

video-prediction-eval

Clockwork Variational Autoencoders — Saxena et al. (2021) (arXiv:2102.09532, 2021)

What this evaluates

Evaluates the ability of generative models to perform long-horizon open-loop video prediction. It probes how well models maintain temporal consistency, preserve object identities, and adapt to varying scene dynamics across diverse visual domains.

Datasets

• MineRL Navigate — total 750000; splits: test (-1)
• KTH Action — total 290000; splits: test (-1)
• GQN Mazes — total 9000000; splits: test (-1)
• Moving MNIST — total 2000000; splits: test (-1)

Metrics

• SSIM — range: [0, 1]
  • Measures perceptual similarity between two images based on luminance, contrast, and structure. Ranges from 0 to 1, where 1 indicates identical images.
• PSNR — range: other
  • Measures the ratio between the maximum possible power of a signal and the power of distorting noise. Higher values indicate better quality.
• LPIPS — range: [0, 1]
  • Computes perceptual similarity by comparing deep feature representations of two images. Lower values indicate higher perceptual similarity.
• FVD (primary) — range: other
  • Measures the distance between the distributions of real and generated video features using a pre-trained Inception-V3 network. Lower values indicate better video quality and temporal coherence.

Input / output format

Input: A sequence of the first 36 frames (64x64 pixels) from a video dataset.

Output: A sequence of predicted future frames (open-loop, without ground truth intermediate frames) matching the evaluation horizon length.

Scoring recipe

def compute_metrics(pred_frames, gt_frames):
    ssim_scores = [ssim(p, g) for p, g in zip(pred_frames, gt_frames)]
    psnr_scores = [psnr(p, g) for p, g in zip(pred_frames, gt_frames)]
    lpips_scores = [lpips(p, g) for p, g in zip(pred_frames, gt_frames)]
    fvd_score = frechet_video_distance([pred_frames], [gt_frames])
    return {
        "SSIM": mean(ssim_scores),
        "PSNR": mean(psnr_scores),
        "LPIPS": mean(lpips_scores),
        "FVD": fvd_score
    }
# Aggregate: average rank across all datasets and metrics for final comparison

Common pitfalls

• Open-loop evaluation requires models to predict without teacher forcing; using ground truth frames as input for subsequent steps invalidates the protocol.
• Rank aggregation combines metrics with opposing optimization directions (SSIM/PSNR higher is better vs. LPIPS/FVD lower is better), which can mask dataset-specific strengths if not normalized correctly.
• Temporal abstraction factors must be tuned to the dataset's frame rate/dynamics; fixed factors may underperform on sequences with different motion speeds.

Evidence (verbatim from paper)

We evaluate the open-loop video predictions under 4 metrics: Structural Similarity index (SSIM, higher is better), Peak Signal-to-Noise Ratio (PSNR, higher is better), Learned Perceptual Image Patch Similarity (LPIPS, lower is better; Zhang et al., 2018), and Frechet Video Distance (FVD, lower is better; Unterthiner et al., 2018). All video predictions are open-loop, meaning that the models only receive the first 36 frames as context input and then predict forward without access to intermediate frames.

Citation

@misc{saxena2021clockwork,
  title={Clockwork Variational Autoencoders},
  author={Saxena et al. (2021)},
  year={2021},
  note={arXiv:2102.09532}
}

• arXiv: 2102.09532