Stream

stream

STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models — Kim et al. (2024) (arXiv:2403.09669, 2024)

What this evaluates

Evaluates the spatial realism and temporal flow consistency of videos generated by video generative models. It decomposes video features into spatial and temporal components using embedding spaces and Fourier transforms to provide length-agnostic, bounded scores that independently measure visual fidelity and motion naturalness.

Datasets

• CATER — total ?; splits: test (-1)
• Kinetics-600 — total ?; splits: test (-1)
• UCF-101 — total ?; splits: test (-1)

Metrics

• STREAM-S — range: [0, 1]
  • Measures spatial quality by analyzing static/background components of video embeddings. Scores decrease proportionally with increasing spatial noise or degradation intensity.
• STREAM-T (primary) — range: [0, 1]
  • Measures temporal flow and naturalness by applying Fast Fourier Transform (FFT) along the temporal axis to isolate motion components. Scores reflect consistency of movement and remain robust to spatial noise.

Input / output format

Input: Pairs of real and generated video clips (typically 16 or 128 frames at 128x128 resolution). Each clip is processed through an image embedding network, then transformed along the temporal axis.

Output: Scalar scores for spatial quality (STREAM-S) and temporal flow (STREAM-T), reported as averages over 5 repeated measurements with standard deviation.

Scoring recipe

def compute_stream(real_videos, fake_videos):
    # 1. Extract frame embeddings using a pre-trained image encoder
    real_emb = [encoder(frames) for frames in real_videos]
    fake_emb = [encoder(frames) for frames in fake_videos]
    # 2. Apply FFT along the temporal axis to decompose features
    real_fft = fft(real_emb, axis='time')
    fake_fft = fft(fake_emb, axis='time')
    # 3. Separate spatial (low temporal freq) and temporal (high temporal freq) components
    spatial_real, temporal_real = decompose_spatial_temporal(real_fft)
    spatial_fake, temporal_fake = decompose_spatial_temporal(fake_fft)
    # 4. Compute bounded distance metrics between real and fake distributions
    stream_s = spatial_distance(spatial_real, spatial_fake)
    stream_t = temporal_distance(temporal_real, temporal_fake)
    return stream_s, stream_t

Common pitfalls

• FVD is unbounded and heavily biased toward spatial quality, making it unreliable for long videos or specific noise types like random translation.
• Sliding-window adaptations (sFVD/sVIS) fail to capture true temporal degradation across full sequences, artificially inflating scores for long videos.
• Metrics must be averaged over 5 repeated measurements to account for variance; single-run scores are not comparable or statistically valid.

Evidence (verbatim from paper)

We assess the capability of STREAM to accurately evaluate spatial and temporal aspects of video data. We employ a series of tests involving synthetic toy data and actual samples generated by video generative models to ensure a comprehensive evaluation of the effectiveness and reliability of the proposed metric in various scenarios. In all experiments, we consider a total of 2,048 real and fake data. The results for all metrics are the average of five repeated measurements.

Citation

@misc{kim2024stream,
  title={STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2403.09669}
}

• arXiv: 2403.09669