Vc Inspector

vc-inspector

VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analy — Shubhashis Roy Dipta et al. (2025) (arXiv:2509.16538, 2025)

What this evaluates

Evaluates the factual accuracy and overall quality of video captions in a reference-free setting. It measures how well a model's predicted quality scores and explanations align with human judgments across diverse video and image domains.

Datasets

• VATEX-EVAL — total 15540; splits: test (15540)
• ActivityNet-FG-Eval — total ?; splits: eval (-1)
• YouCook2-FG-Eval — total ?; splits: eval (-1)
• Flickr8K-Expert — total 17000; splits: test (17000)
• Flickr8K-CF — total 48000; splits: test (48000)

Metrics

• Kendall's correlation ($\tau_b$) (primary) — range: [-1, 1] (reported as 0–100% in tables)
  • Measures the rank correlation between predicted caption quality scores and human-annotated scores. Adjusts for ties in both predicted and gold scores to provide a robust ordinal correlation estimate.
• Spearman's rank correlation ($\rho$) — range: [-1, 1] (reported as 0–100% in tables)
  • Measures the monotonic relationship between predicted and human-annotated quality scores based on rank differences. Computed as 1 - (6 * sum(d_i^2)) / (n * (n^2 - 1)).

Input / output format

Input: Video frames (uniformly sampled to 32 frames, resized to 224x224) or a single image, paired with a candidate caption.

Output: A quality score on a 1–5 scale and a textual explanation identifying factual errors in the caption.

Scoring recipe

def compute_kendall_tau_b(pred_scores, gold_scores):
    n = len(pred_scores)
    concordant = discordant = 0
    for i in range(n):
        for j in range(i + 1, n):
            diff_p = pred_scores[i] - pred_scores[j]
            diff_g = gold_scores[i] - gold_scores[j]
            if diff_p * diff_g > 0: concordant += 1
            elif diff_p * diff_g < 0: discordant += 1
    n0 = n * (n - 1) / 2
    tau_b = (concordant - discordant) / n0
    return tau_b * 100  # Paper reports as percentage

Common pitfalls

• The synthetic evaluation datasets (ActivityNet-FG-Eval, YouCook2-FG-Eval) use LLM-generated pseudo-scores, not direct human annotations; human alignment is only verified on VATEX-EVAL and Flickr8K.
• Correlation scores measure ranking alignment with human judgments, not absolute score accuracy; higher values indicate better relative ordering of caption quality.
• The metric is explicitly designed for the reference-free setting, but results are often compared against reference-based baselines, which can mislead readers about its actual input requirements.

Evidence (verbatim from paper)

Following prior work, we report both Kendall’s correlation ($\tau_{b}$) and Spearman’s rank correlation ($\rho$). In the caption evaluation literature, the effectiveness of evaluation metrics is typically assessed by measuring their correlation to ground truth.

Citation

@misc{roydipta2025vcinspector,
  title={VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analy},
  author={Shubhashis Roy Dipta et al. (2025)},
  year={2025},
  note={arXiv:2509.16538}
}

• arXiv: 2509.16538