Vggsounder Eval

vggsounder-eval

VGGSounder: Audio-Visual Evaluations for Foundation Models — Zverev et al. (2025) (arXiv:2508.08237, 2025)

What this evaluates

Evaluates audio-visual foundation and embedding models on multi-label video classification, probing their ability to recognize sound and visual events across different input modalities. It specifically measures modality alignment, unimodal versus multimodal performance, and susceptibility to distraction from irrelevant background audio or static visuals.

Datasets

• VGGSounder — total ?; splits: test (-1)

Metrics

• Subset Accuracy — range: percent
  • Fraction of samples where the predicted label set exactly matches the ground-truth label set.
• F1-score (primary) — range: percent
  • Harmonic mean of precision and recall for multi-label classification, computed with micro-aggregation to balance class contributions.
• Hit — range: percent
  • Fraction of samples where at least one predicted label is present in the ground-truth label set.
• Modality Confusion ($\mu$) — range: percent
  • Percentage of samples correctly classified given a unimodal input but misclassified when both audio and visual inputs are provided simultaneously.

Input / output format

Input: Per-sample audio stream, video stream, or both. For foundation models, inputs are paired with a prompt containing the full list of 309 target classes.

Output: For embedding models: top-k ranked class predictions. For foundation models: an unordered set of predicted class labels of varying size.

Scoring recipe

def compute_metrics(preds, golds):
    hits, tp, fp, fn, subset = 0, 0, 0, 0, 0
    for p, g in zip(preds, golds):
        p_set, g_set = set(p), set(g)
        if p_set & g_set: hits += 1
        tp += len(p_set & g_set)
        fp += len(p_set - g_set)
        fn += len(g_set - p_set)
        if p_set == g_set: subset += 1
    n = len(preds)
    hit = hits / n
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    sub_acc = subset / n
    return hit, f1, sub_acc

Common pitfalls

• Embedding models (top-k predictions) and foundation models (unordered variable-size sets) produce metrics that are not directly comparable due to different output formats.
• Using the original VGGSound labels without VGGSounder's per-sample modality annotations leads to false positives and underestimates audio performance, as background sounds and unaligned cues are not properly accounted for.
• Models often rely exclusively on visual inputs; evaluating only multimodal setups masks severe unimodal audio degradation.

Evidence (verbatim from paper)

Subset accuracy compares the predicted label set to the ground-truth label set and reports the fraction of samples for which they match. This is our strictest metric. F1-score is the harmonic mean of precision and recall. It is strictly larger than the subset accuracy. Hit reports the fraction of samples for which any of the predicted labels are part of the ground-truth label set. This is the most lenient metric which is strictly larger than the F1-score.

Citation

@misc{zverev2025vggsounder,
  title={VGGSounder: Audio-Visual Evaluations for Foundation Models},
  author={Zverev et al. (2025)},
  year={2025},
  note={arXiv:2508.08237}
}

• arXiv: 2508.08237