# Video Prediction Eval

> Evaluates the ability of generative models to perform long-horizon open-loop video prediction. It probes how well models maintain temporal consistency, preserve object identities, and adapt to varying scene dynamics across diverse visual domains. Use when the user wants to benchmark on MineRL Navigate, KTH Action, GQN Mazes, Moving MNIST, or asks about evaluating this task. Reports FVD.

- Skill: `qhjqhj00/video-prediction-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/video-prediction-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/video-prediction-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/video-prediction-eval

---


# video-prediction-eval

> Clockwork Variational Autoencoders — Saxena et al. (2021) (arXiv:2102.09532, 2021)

## What this evaluates

Evaluates the ability of generative models to perform long-horizon open-loop video prediction. It probes how well models maintain temporal consistency, preserve object identities, and adapt to varying scene dynamics across diverse visual domains.

## Datasets

- **MineRL Navigate** — total 750000; splits: test (-1)
- **KTH Action** — total 290000; splits: test (-1)
- **GQN Mazes** — total 9000000; splits: test (-1)
- **Moving MNIST** — total 2000000; splits: test (-1)

## Metrics

- `SSIM` — range: [0, 1]
  - Measures perceptual similarity between two images based on luminance, contrast, and structure. Ranges from 0 to 1, where 1 indicates identical images.
- `PSNR` — range: other
  - Measures the ratio between the maximum possible power of a signal and the power of distorting noise. Higher values indicate better quality.
- `LPIPS` — range: [0, 1]
  - Computes perceptual similarity by comparing deep feature representations of two images. Lower values indicate higher perceptual similarity.
- `FVD` **(primary)** — range: other
  - Measures the distance between the distributions of real and generated video features using a pre-trained Inception-V3 network. Lower values indicate better video quality and temporal coherence.

## Input / output format

**Input**: A sequence of the first 36 frames (64x64 pixels) from a video dataset.

**Output**: A sequence of predicted future frames (open-loop, without ground truth intermediate frames) matching the evaluation horizon length.

## Scoring recipe

```python
def compute_metrics(pred_frames, gt_frames):
    ssim_scores = [ssim(p, g) for p, g in zip(pred_frames, gt_frames)]
    psnr_scores = [psnr(p, g) for p, g in zip(pred_frames, gt_frames)]
    lpips_scores = [lpips(p, g) for p, g in zip(pred_frames, gt_frames)]
    fvd_score = frechet_video_distance([pred_frames], [gt_frames])
    return {
        "SSIM": mean(ssim_scores),
        "PSNR": mean(psnr_scores),
        "LPIPS": mean(lpips_scores),
        "FVD": fvd_score
    }
# Aggregate: average rank across all datasets and metrics for final comparison
```

## Common pitfalls

- Open-loop evaluation requires models to predict without teacher forcing; using ground truth frames as input for subsequent steps invalidates the protocol.
- Rank aggregation combines metrics with opposing optimization directions (SSIM/PSNR higher is better vs. LPIPS/FVD lower is better), which can mask dataset-specific strengths if not normalized correctly.
- Temporal abstraction factors must be tuned to the dataset's frame rate/dynamics; fixed factors may underperform on sequences with different motion speeds.

## Evidence (verbatim from paper)

> We evaluate the open-loop video predictions under 4 metrics: Structural Similarity index (SSIM, higher is better), Peak Signal-to-Noise Ratio (PSNR, higher is better), Learned Perceptual Image Patch Similarity (LPIPS, lower is better; Zhang et al., 2018), and Frechet Video Distance (FVD, lower is better; Unterthiner et al., 2018). All video predictions are open-loop, meaning that the models only receive the first 36 frames as context input and then predict forward without access to intermediate frames.

## Citation

```bibtex
@misc{saxena2021clockwork,
  title={Clockwork Variational Autoencoders},
  author={Saxena et al. (2021)},
  year={2021},
  note={arXiv:2102.09532}
}
```

- arXiv: 2102.09532