Cine Tech Bench Eval

cine-tech-bench-eval

CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation — Wang et al. (2025) (arXiv:2505.15145, 2025)

What this evaluates

Evaluates multimodal large language models and video generation models on fine-grained cinematographic understanding (shot scale, angle, composition, camera movement, lighting, color, focal length) and camera movement generation from video clips.

Datasets

• CineTechBench — total 720; splits: test (720); repo https://github.com/PRIS-CV/CineTechBench

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions for static and dynamic cinematographic dimensions.
• BLEU@4 — range: percent
  • 4-gram overlap between generated description and ground truth.
• METEOR — range: percent
  • Harmonic mean of unigram precision and recall with synonym/stem matching.
• ROUGE-L — range: percent
  • Longest common subsequence overlap between generated and reference descriptions.
• CAPability-F1 — range: [0, 1]
  • F1-score derived from hit rate (HR), average precision (AP), and average recall (AR) based on the benchmark's cinematographic taxonomy.
• RotErr — range: other
  • Rotation error between generated and original camera trajectory, estimated via MonST3R.
• TransErr — range: other
  • Translation error between generated and original camera trajectory, estimated via MonST3R.
• CamMC — range: other
  • Camera motion consistency metric quantifying trajectory deviation.
• CLIP-IS — range: [0, 1]
  • CLIP-based frame similarity score measuring visual consistency between generated and original frames.

Input / output format

Input: Understanding: image or video clip paired with a question about a specific cinematographic technique. Generation: first frame, optionally last frame, and textual description of the desired camera movement.

Output: Understanding: text answer (multiple-choice selection or open-ended description). Generation: a video clip.

Scoring recipe

def score_understanding(preds, golds):
    acc = sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)
    bleu = compute_bleu(golds, preds)
    meteor = compute_meteor(golds, preds)
    rouge = compute_rouge_l(golds, preds)
    cap_f1 = compute_cap_f1(golds, preds)  # HR, AP, AR based on taxonomy
    return {'accuracy': acc, 'BLEU@4': bleu, 'METEOR': meteor, 'ROUGE-L': rouge, 'CAPability-F1': cap_f1}

def score_generation(gen_videos, orig_videos):
    traj_gen = estimate_trajectory(gen_videos)  # MonST3R
    traj_orig = estimate_trajectory(orig_videos)
    rot_err = compute_rotation_error(traj_gen, traj_orig)
    trans_err = compute_translation_error(traj_gen, traj_orig)
    cammc = compute_cam_mc(traj_gen, traj_orig)
    clip_is = compute_clip_similarity(gen_videos, orig_videos)
    return {'RotErr': rot_err, 'TransErr': trans_err, 'CamMC': cammc, 'CLIP-IS': clip_is}

Common pitfalls

• Models frequently misclassify lighting directions (e.g., confusing Side Light with Back Light) and struggle with complex camera rotations.
• Reference-based n-gram metrics (BLEU/ROUGE) poorly capture fine-grained cinematographic descriptions; CAPability metrics are strongly recommended for generation evaluation.
• Video generation models often reverse rotation direction or fail to generate roll movements despite achieving correct translation errors.

Evidence (verbatim from paper)

For question-answering tasks, we report overall accuracy as well as accuracy broken down by each cinematography dimension. For description generation tasks, we use four reference-based metrics. Three of these—BLEU, METEOR, and ROUGE—are based on n-gram overlap. To address this, we additionally incorporate evaluation metrics from the CAPability benchmark based on our taxonomy, which reliably assess both the correctness and thoroughness of MLLM-generated descriptions using hit rate (HR), average precision (AP), average recall (AR) and F1-score. In this section, we use video generation models to reconstruct the camera movement in the original film clip by inputting the first frame, the last frame (if applicable), and textual description. Following prior work, we quantify trajectory similarity between the generated and the original video clips via three metrics: rotation error (RotErr), translation error (TransErr), CamMC. We use MonST3R to estimate the camera trajectory of the generated and original movie clip. Finally, we also report a CLIP-based frame similarity score (CLIP-IS) to capture visual consistency.

Citation

@misc{wang2025cine techbench,
  title={CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.15145}
}

• arXiv: 2505.15145