# Vefx Bench Eval

> vefx-bench-eval

- Skill: `qhjqhj00/vefx-bench-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/vefx-bench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/vefx-bench-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/vefx-bench-eval

---


# vefx-bench-eval

> VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects — Gao et al. (2026) (arXiv:2604.16272, 2026)

## What this evaluates

Evaluates video editing reward models and VLM judges on their ability to align with human preferences across three dimensions: instruction following, rendering quality, and edit exclusivity. It tests both global score correlation and local pairwise preference consistency within candidate sets.

## Datasets

- **VEFX-Bench** — total 5049; splits: train (4200), test (849)

## Metrics

- `SRCC` **(primary)** — range: [0, 1]
  - Spearman Rank-Order Correlation Coefficient between raw model predictions and human scores.
- `KRCC` — range: [0, 1]
  - Kendall Rank-Order Correlation Coefficient between raw model predictions and human scores.
- `PLCC` — range: [0, 1]
  - Pearson Linear Correlation Coefficient computed after applying a standard 4-parameter logistic calibration to raw predictions.
- `RMSE` — range: other
  - Root Mean Squared Error between calibrated predictions and human scores.
- `Pairwise Accuracy` — range: [0, 1]
  - Fraction of correctly ordered candidate pairs within each group. Ties in ground truth count as correct; ties in prediction score 0.5.

## Input / output format

**Input**: Source video, editing instruction, and edited output video.

**Output**: Scores on a 1–4 rubric for Instruction Following (IF), Rendering Quality (RQ), and Edit Exclusivity (EE), or an overall score (mean of the three dimensions).

## Scoring recipe

```python
import scipy.stats as stats
import numpy as np

def compute_metrics(preds, golds):
    srcc = stats.spearmanr(preds, golds).correlation
    krcc = stats.kendalltau(preds, golds).correlation
    # PLCC/RMSE require 4-param logistic calibration per Sec 15.1
    rmse = np.sqrt(np.mean((preds - golds)**2))
    return srcc, krcc, rmse

def pairwise_accuracy(groups, preds, golds):
    correct, total = 0, 0
    for g in groups:
        for i, j in pairs_in(g):
            if golds[i] == golds[j]: correct += 1
            elif (preds[i] > preds[j]) == (golds[i] > golds[j]): correct += 1
            elif preds[i] == preds[j]: correct += 0.5
            total += 1
    return correct / total
```

## Common pitfalls

- PLCC and RMSE must be computed after 4-parameter logistic calibration, not on raw predictions.
- Pairwise Accuracy is strictly computed within groups sharing the same source video and instruction, not globally across all candidates.
- The overall human score is the arithmetic mean of IF, RQ, and EE scores, not a separately annotated dimension.

## Evidence (verbatim from paper)

> Our primary evaluation follows standard IQA/VQA protocol. We report Spearman Rank-Order Correlation Coefficient (SRCC), Kendall Rank-Order Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Squared Error (RMSE) in [Section˜5.2]. SRCC and KRCC are computed on raw predictions, while PLCC and RMSE are computed after the standard four-parameter logistic calibration. We complement these global correlation metrics with a group-wise preference metric, Pairwise Accuracy, in [Section˜5.3].

## Citation

```bibtex
@misc{gao2026vefxbench,
  title={VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2604.16272}
}
```

- arXiv: 2604.16272