Oscbench Eval

oscbench-eval

OSCBench: Benchmarking Object State Change in Text-to-Video Generation — Han et al. (2026) (arXiv:2603.11698, 2026)

What this evaluates

This benchmark probes a text-to-video model's ability to accurately render and maintain object state transformations (e.g., peeling, slicing) over time. It additionally measures semantic adherence to prompts, scene consistency, and overall perceptual quality to diagnose temporal coherence and physical realism in generated videos.

Datasets

• OSCBench — total 140; splits: test (140)

Metrics

• state-change accuracy (primary) — range: [1, 5]
  • Average score on a 1–5 Likert scale indicating whether the object reaches the correct target state implied by the prompt.
• state-change consistency — range: [1, 5]
  • Average score on a 1–5 Likert scale assessing whether the transformation unfolds smoothly over time without abrupt jumps or unnatural appearances.
• Semantic Adherence — range: [1, 5]
  • Average score on a 1–5 Likert scale measuring alignment of subject, object, and action entities from the prompt in the generated video.
• Scene Alignment — range: [1, 5]
  • Average score on a 1–5 Likert scale evaluating whether the global environment and background context match the prompt description.
• Perceptual Quality — range: [1, 5]
  • Average score on a 1–5 Likert scale reflecting overall visual impression, including realism and aesthetic composition.

Input / output format

Input: A text prompt specifying an object state change scenario, paired with the generated video to be evaluated.

Output: A discrete integer score from 1 to 5 for each evaluation dimension.

Scoring recipe

def compute_metric(scores_1_5):
    # scores_1_5: list of 1-5 ratings from 3 human evaluators or MLLM CoT output
    # For humans: average across 3 independent ratings
    # For MLLMs: score derived from CoT reasoning (criteria grounding -> evidence extraction -> score decision)
    return sum(scores_1_5) / len(scores_1_5)

Common pitfalls

• Using coarse text-video similarity models (e.g., CLIP, ViCLIP) instead of reasoning-based evaluators, as they fail to capture fine-grained object state transitions.
• Evaluating all generated videos exhaustively with humans is prohibitively costly; the benchmark uses a representative sampling strategy (1 prompt per scenario) rather than full coverage.
• MLLM evaluators must be prompted with Chain-of-Thought reasoning; without it, they are easily distracted by salient but irrelevant visual details and miss temporal state changes.

Evidence (verbatim from paper)

state-change accuracy, which measures whether the object reaches the correct target state implied by the prompt (e.g., a whole apple becoming sliced), and state-change consistency, which assesses whether the transformation unfolds smoothly and coherently over time, without abrupt jumps or unnatural object appearances or unexplained appearance or disappearance of object parts.

Citation

@misc{han2026oscbench,
  title={OSCBench: Benchmarking Object State Change in Text-to-Video Generation},
  author={Han et al. (2026)},
  year={2026},
  note={arXiv:2603.11698}
}

• arXiv: 2603.11698