Video Captioning Eval

video-captioning-eval

Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions — Li et al. (2026) (arXiv:2602.13013, 2026)

What this evaluates

Evaluates fine-grained audiovisual captioning quality, attribute-level instruction following, and downstream reasoning capabilities like QA and temporal grounding.

Datasets

• video-SALMONN-2 — total ?; splits: test (-1)
• UGC-VideoCap — total ?; splits: test (-1)
• VDC — total ?; splits: test (-1)
• VidCapBench-AE — total ?; splits: test (-1)
• Daily-Omni — total ?; splits: test (-1)
• World-Sense — total ?; splits: test (-1)
• Charades-STA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered QA questions or correctly followed attribute instructions.
• mIoU — range: [0, 1]
  • Mean Intersection over Union between predicted and ground-truth temporal segments.
• R1@τ — range: [0, 1]
  • Recall at IoU threshold τ, measuring the fraction of predictions where IoU ≥ τ.
• Miss/Hall/Total — range: [0, 1]
  • Rates of missing details, hallucinations, and total errors in captions, respectively.
• Acc/Pre/Cov/Con — range: [0, 100]
  • Accuracy, precision, coverage, and conciseness scores for text-to-video generation captions, scaled by 100.

Input / output format

Input: Video frames/audio, optionally accompanied by attribute instructions or QA prompts.

Output: Text captions (optionally timestamped and attribute-structured), QA answers, or temporal start/end timestamps.

Scoring recipe

def score(predictions, gold):
    acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    ious = [iou(p.start, p.end, g.start, g.end) for p, g in zip(predictions, gold)]
    miou = sum(ious) / len(ious)
    r1_tau = sum(1 for i in ious if i >= tau) / len(ious)
    miss = sum(1 for p in predictions if p.flags.missing) / len(predictions)
    hall = sum(1 for p in predictions if p.flags.hallucination) / len(predictions)
    return acc, miou, r1_tau, miss, hall

Common pitfalls

• LLM-based judges (GPT-4o, Gemini-2.5-Pro) are used for scoring, which may introduce model-specific biases.
• Hallucination metrics penalize unannotated but relevant details (e.g., on-screen text) as errors.
• Temporal grounding relies on caption-only prompts to a fixed QA model, which may not reflect end-to-end model performance.

Evidence (verbatim from paper)

We evaluate ASID-Captioner on seven benchmarks that offer complementary perspectives on caption quality. ... Start and end timestamps of the queried moment are predicted using only the caption, and we report mIoU and recall at IoU thresholds (R1@$ au$).

Citation

@misc{li2026towards,
  title={Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2602.13013}
}

• arXiv: 2602.13013