Avid Eval

avid-eval

AVID: A Benchmark for Omni-Modal Audio-Visual Inconsistency Understanding via Agent-Driven Construction — Chen et al. (2026) (arXiv:2604.13593, 2026)

What this evaluates

Evaluates a model's ability to detect, classify, and temporally ground audio-visual inconsistencies in long-form videos, as well as generate causal explanations for cross-modal mismatches across eight fine-grained categories.

Datasets

• AVID — total 11200; splits: train (9639), test (1561)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union for temporal grounding, calculated as the average IoU across all predicted inconsistency segments compared to ground-truth timestamps.
• BLEU-4 — range: [0, 1]
  • 4-gram BLEU score measuring the n-gram overlap between generated causal explanations or reasoning text and reference annotations.

Input / output format

Input: Full-length videos or temporally segmented clips containing synchronized audio and visual streams.

Output: Predicted inconsistency labels (consistent/inconsistent), classification into one of 8 fine-grained categories, temporal boundaries (start/end timestamps) for each event, and a textual causal explanation.

Scoring recipe

def compute_mIoU(pred_segments, gold_segments):
    ious = []
    for pred, gold in zip(pred_segments, gold_segments):
        intersection = max(0, min(pred.end, gold.end) - max(pred.start, gold.start))
        union = max(pred.end, gold.end) - min(pred.start, gold.start)
        ious.append(intersection / union if union > 0 else 0.0)
    return sum(ious) / len(ious) if ious else 0.0

def compute_bleu4(predictions, references):
    return nltk.bleu_score.corpus_bleu([[ref] for ref in references], predictions, weights=(0.25, 0.25, 0.25, 0.25))

Common pitfalls

• Inconsistency patterns are class-dependent (e.g., lip-sync vs. semantic vs. environmental), so models must adapt reasoning per segment type rather than using a single heuristic.
• Evaluation operates at both video and segment levels; confusing segment-level temporal grounding with video-level classification leads to inflated or deflated scores.
• Agent-driven construction may introduce synthesis artifacts that models could reverse-engineer instead of performing true cross-modal reasoning.

Evidence (verbatim from paper)

enables evaluation of detection, classification, and reasoning tasks with precise temporal annotations and causal explanations, revealing significant limitations in state-of-the-art omni-modal models—particularly in temporal grounding (mIoU: 36.1% vs. 26.2%) and segment-level reasoning (2.8× BLEU-4 improvement over base models)

Citation

@misc{chen2026avid,
  title={AVID: A Benchmark for Omni-Modal Audio-Visual Inconsistency Understanding via Agent-Driven Construction},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2604.13593}
}

• arXiv: 2604.13593