Leum Vl Eval

leum-vl-eval

Leum-VL Technical Report — He et al. (2026) (arXiv:2603.20354, 2026)

What this evaluates

Evaluates a video-language model's capacity for structured video understanding across six professional dimensions (subject, aesthetics, camera language, editing, narrative, dissemination) while maintaining general multimodal capabilities. It probes timeline-grounded reasoning, temporal localization, and document/OCR comprehension.

Datasets

• FeedBench — total ?; splits: test (-1); repo https://github.com/leum-team/leum-v1
• Open Benchmarks (Video-MME, MVBench, MMBench-EN, etc.) — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard benchmark accuracy calculated as the percentage of correctly predicted answers or dimension scores. For FeedBench, an LLM judge (DeepSeek-V3) evaluates structured outputs against expert-sensitive label boundaries.

Input / output format

Input: Video content sampled at 4 FPS with a maximum of 768 frames, accompanied by a text prompt/question. Total input is capped at 50K tokens per video.

Output: Text response answering the prompt or providing structured labels for the video dimensions.

Scoring recipe

def compute_accuracy(predictions, gold, use_llm_judge=False):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if use_llm_judge:
            verdict = llm_judge_evaluate(pred, gold_item)  # DeepSeek-V3
            correct += verdict
        else:
            correct += (pred == gold_item)
    return (correct / len(gold)) * 100

Common pitfalls

• FeedBench relies on DeepSeek-V3 as an LLM judge rather than deterministic ground truth, which may introduce evaluation variance or bias.
• Fixed frame sampling (FPS=4, max 768 frames) and token limits may truncate fast-paced or long videos, affecting temporal grounding metrics.
• Cross-benchmark comparisons mix official reported values with local reproductions (marked †), complicating direct performance baselines.

Evidence (verbatim from paper)

On RefineShot, we improve overall accuracy by +4.2 pp (56.9 vs. 52.7), with the largest gains on Shot Framing (+13.4pp), Camera Angle (+10.8pp), and Lighting Type (+6.4pp) dimensions that correspond directly to the SV6D schema. Eval: FPS=4, max 768 frames, max 50K tokens/video. Judge: DeepSeek-V3 (DeepSeek-AI, 2025).

Citation

@misc{he2026leumvl,
  title={Leum-VL Technical Report},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2603.20354}
}

• arXiv: 2603.20354