Vefx Bench Eval

vefx-bench-eval

VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects — Gao et al. (2026) (arXiv:2604.16272, 2026)

What this evaluates

Evaluates video editing reward models and VLM judges on their ability to align with human preferences across three dimensions: instruction following, rendering quality, and edit exclusivity. It tests both global score correlation and local pairwise preference consistency within candidate sets.

Datasets

• VEFX-Bench — total 5049; splits: train (4200), test (849)

Metrics

• SRCC (primary) — range: [0, 1]
  • Spearman Rank-Order Correlation Coefficient between raw model predictions and human scores.
• KRCC — range: [0, 1]
  • Kendall Rank-Order Correlation Coefficient between raw model predictions and human scores.
• PLCC — range: [0, 1]
  • Pearson Linear Correlation Coefficient computed after applying a standard 4-parameter logistic calibration to raw predictions.
• RMSE — range: other
  • Root Mean Squared Error between calibrated predictions and human scores.
• Pairwise Accuracy — range: [0, 1]
  • Fraction of correctly ordered candidate pairs within each group. Ties in ground truth count as correct; ties in prediction score 0.5.

Input / output format

Input: Source video, editing instruction, and edited output video.

Output: Scores on a 1–4 rubric for Instruction Following (IF), Rendering Quality (RQ), and Edit Exclusivity (EE), or an overall score (mean of the three dimensions).

Scoring recipe

import scipy.stats as stats
import numpy as np

def compute_metrics(preds, golds):
    srcc = stats.spearmanr(preds, golds).correlation
    krcc = stats.kendalltau(preds, golds).correlation
    # PLCC/RMSE require 4-param logistic calibration per Sec 15.1
    rmse = np.sqrt(np.mean((preds - golds)**2))
    return srcc, krcc, rmse

def pairwise_accuracy(groups, preds, golds):
    correct, total = 0, 0
    for g in groups:
        for i, j in pairs_in(g):
            if golds[i] == golds[j]: correct += 1
            elif (preds[i] > preds[j]) == (golds[i] > golds[j]): correct += 1
            elif preds[i] == preds[j]: correct += 0.5
            total += 1
    return correct / total

Common pitfalls

• PLCC and RMSE must be computed after 4-parameter logistic calibration, not on raw predictions.
• Pairwise Accuracy is strictly computed within groups sharing the same source video and instruction, not globally across all candidates.
• The overall human score is the arithmetic mean of IF, RQ, and EE scores, not a separately annotated dimension.

Evidence (verbatim from paper)

Our primary evaluation follows standard IQA/VQA protocol. We report Spearman Rank-Order Correlation Coefficient (SRCC), Kendall Rank-Order Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Squared Error (RMSE) in [Section˜5.2]. SRCC and KRCC are computed on raw predictions, while PLCC and RMSE are computed after the standard four-parameter logistic calibration. We complement these global correlation metrics with a group-wise preference metric, Pairwise Accuracy, in [Section˜5.3].

Citation

@misc{gao2026vefxbench,
  title={VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2604.16272}
}

• arXiv: 2604.16272