Audio Visual Gfsl Eval

audio-visual-gfsl-eval

Text-to-feature diffusion for audio-visual few-shot learning — Mercea et al. (2023) (arXiv:2309.03869, 2023)

What this evaluates

Evaluates audio-visual few-shot video classification by measuring how well models generalize to novel classes with limited training examples (1, 5, 10-shot). It reports both generalised few-shot learning (HM) and standard few-shot learning (FSL) accuracy to assess bias towards base classes.

Datasets

• VGGSound-FSL — total ?; splits: test (-1); repo https://github.com/ExplainableML/AVDIFF-GFSL
• UCF-FSL — total ?; splits: test (-1); repo https://github.com/ExplainableML/AVDIFF-GFSL
• ActivityNet-FSL — total ?; splits: test (-1); repo https://github.com/ExplainableML/AVDIFF-GFSL

Metrics

• HM (primary) — range: percent
  • Harmonic mean of the mean class accuracies for base and novel classes. HM = 2 * (Acc_base * Acc_novel) / (Acc_base + Acc_novel).
• FSL — range: percent
  • Mean class accuracy computed only on the test subset of novel classes.

Input / output format

Input: Pre-extracted audio and visual features from pre-trained classification networks, processed via cross-modal attention, and conditioned on text embeddings (word2vec) or class prototypes.

Output: Predicted class labels for video clips, used to compute per-class accuracy on base and novel sets.

Scoring recipe

def compute_metrics(predictions, gold, base_classes, novel_classes):
    acc_base = mean([accuracy(predictions[c], gold[c]) for c in base_classes])
    acc_novel = mean([accuracy(predictions[c], gold[c]) for c in novel_classes])
    FSL = acc_novel
    HM = 2 * (acc_base * acc_novel) / (acc_base + acc_novel)
    return HM, FSL

Common pitfalls

• Confusing HM (generalised FSL) with FSL (novel-only accuracy); HM explicitly penalizes models biased towards base classes.
• ActivityNet-FSL contains very long videos requiring evaluation-time trimming to a maximum of 300 features, which significantly impacts temporal modeling compared to other datasets.
• Calibrated stacking is used to reduce base-class bias, which can alter the expected relationship between HM, novel accuracy (N), and FSL scores.

Evidence (verbatim from paper)

The harmonic mean (HM) of the mean class accuracies for base and novel classes are reported for GFSL. For the FSL performance, only the test subset of the novel classes is considered.

Citation

@misc{mercea2023texttofeaturediffusion,
  title={Text-to-feature diffusion for audio-visual few-shot learning},
  author={Mercea et al. (2023)},
  year={2023},
  note={arXiv:2309.03869}
}

• arXiv: 2309.03869