# Vivd 10m Eval

> Evaluates video editing models on local, entity-level modifications (addition, modification, deletion) by measuring background preservation, text alignment, temporal consistency, and visual quality. Use when the user wants to benchmark on VIVID-10M-Eval, or asks about evaluating this task. Reports Text Alignment (TA).

- Skill: `qhjqhj00/vivd-10m-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/vivd-10m-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/vivd-10m-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/vivd-10m-eval

---


# vivd-10m-eval

> VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing — Hu et al. (2024) (arXiv:2411.15260, 2024)

## What this evaluates

Evaluates video editing models on local, entity-level modifications (addition, modification, deletion) by measuring background preservation, text alignment, temporal consistency, and visual quality.

## Datasets

- **VIVID-10M-Eval** — total ?; splits: test (-1)

## Metrics

- `Background Preservation (BP)` — range: other
  - L1 distance between the original and edited videos in non-editing regions.
- `Text Alignment (TA)` **(primary)** — range: other
  - CLIP-score of the edited region.
- `Temporal Consistency (TC)` — range: [-1, 1]
  - Cosine similarity between consecutive frames in the CLIP-Image feature space.
- `Visual Quality (VQ)` — range: [0, 1]
  - Human-annotated win rate reflecting realness and aesthetics.

## Input / output format

**Input**: Input video frames, text caption/prompt, and a spatial mask defining the local editing region.

**Output**: Edited video frames matching the input resolution and frame count.

## Scoring recipe

```python
def compute_metrics(original_video, edited_video, mask, text_caption, clip_model):
    # Background Preservation (lower is better)
    bp = np.mean(np.abs(original_video - edited_video) * (1 - mask))
    # Text Alignment (higher is better)
    ta = clip_model.score(edited_region, text_caption)
    # Temporal Consistency (higher is better)
    tc = cosine_similarity(clip_model.encode(frame_t), clip_model.encode(frame_t+1))
    # Visual Quality (win rate from user study)
    vq = (wins + draws) / total_votes
    return bp, ta, tc, vq
```

## Common pitfalls

- BP is a distance metric where lower values indicate better performance, which is counterintuitive for typical accuracy metrics.
- TC measures semantic consistency via CLIP features rather than pixel-level stability, so it may overlook visual jitter or flicker.
- User study results are reported as 'win-or-draw' rates rather than strict win rates, affecting direct comparison with other benchmarks.

## Evidence (verbatim from paper)

> Background Preservation (BP): the L1 distance between the original and edited videos in non-editing regions. Text Alignment (TA): the CLIP-score*[[37], [10]]* of the edited region. Temporal Consistency (TC): the cosine similarity between consecutive frames in the CLIP-Image feature space*[[37]]*.

## Citation

```bibtex
@misc{hu2024vivid10m,
  title={VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing},
  author={Hu et al. (2024)},
  year={2024},
  note={arXiv:2411.15260}
}
```

- arXiv: 2411.15260