Waver Bench Eval

waver-bench-eval

Waver: Wave Your Way to Lifelike Video Generation — Zhang et al. (2025) (arXiv:2508.15761, 2025)

What this evaluates

Evaluates a video generation model's capacity to synthesize physically coherent motion, high-fidelity visuals, and strict adherence to textual or image prompts across diverse scenarios.

Datasets

• Waver-Bench 1.0 — total 304; splits: test (304)
• Hermes Motion Testset — total 96; splits: test (96)

Metrics

• Human Preference Win Rate (primary) — range: [0, 1]
  • Proportion of human raters who prefer the model's generated video over a competitor's in side-by-side comparisons, calculated as (wins + 0.5 * ties) / total_votes. Aggregated across Motion Quality, Visual Quality, and Prompt Following dimensions.
• motion quality — range: percent
  • Human-rated category assessing physical realism and coherence of movement, including Action, Interaction, and Distortion sub-dimensions.

Input / output format

Input: Text prompt (T2V) or reference image + text prompt (I2V)

Output: Generated video clip

Scoring recipe

def compute_win_rate(wins, losses, ties):
    total = wins + losses + ties
    if total == 0:
        return 0.0
    return (wins + 0.5 * ties) / total

Common pitfalls

• Side-by-side human comparisons are susceptible to left/right position bias.
• Benchmarks are proprietary and not publicly released, limiting independent reproduction.
• Artificial Analysis Elo scores depend on user-submitted prompts, making rankings non-deterministic and sensitive to prompt distribution.

Evidence (verbatim from paper)

On these benchmarks, we conducted extensive human evaluations to assess our model’s performance against leading competitors (Veo3, Kling 2.0, and Wan 2.1 14B). In a side-by-side comparison format, human raters were shown videos generated by our model (Waver) and a competitor, then asked to judge which video was superior. The judgment was based on the comprehensive criteria defined above, including motion quality, visual quality, and prompt following.

Citation

@misc{zhang2025waver,
  title={Waver: Wave Your Way to Lifelike Video Generation},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2508.15761}
}

• arXiv: 2508.15761