Vggsound Continual Eval

vggsound-continual-eval

Class-Incremental Grouping Network for Continual Audio-Visual Learning — Mo et al. (2023) (arXiv:2309.05281, 2023)

What this evaluates

Evaluates a model's ability to perform continual audio-visual classification across sequential tasks without catastrophic forgetting. It measures how well the model retains performance on previously learned categories while learning new ones, across audio, visual, and cross-modal fusion settings.

Datasets

• VGGSound-Instruments — total 32000; splits: train (-1), val (-1), test (-1)
• VGGSound-100 — total 97000; splits: train (-1), val (-1), test (-1)
• VGG-Sound Source — total 150000; splits: train (-1), val (-1), test (-1)

Metrics

• Average accuracy (primary) — range: percent
  • The mean classification accuracy across all T sequential tasks. Calculated as the sum of per-task test accuracies divided by T.
• Forgetting — range: percent
  • Measures the performance drop on previously learned tasks after training on subsequent tasks. Typically computed as the average decrease in accuracy from each task's peak performance to its final accuracy.

Input / output format

Input: Paired video clips (10s duration, frames resized to 224×224) and audio segments (3s duration, 22050Hz sample rate converted to 257×300 log spectrograms via STFT). Each instance corresponds to a single audio-visual category.

Output: Class label predictions for three modalities: audio-only, visual-only, and audio-visual fusion.

Scoring recipe

def compute_metrics(task_accuracies, initial_accuracies):
    # task_accuracies: list of final test accuracies per task
    # initial_accuracies: list of peak accuracies achieved per task during training
    avg_acc = sum(task_accuracies) / len(task_accuracies)
    forgetting = sum(max(0, init_acc - curr_acc) for init_acc, curr_acc in zip(initial_accuracies, task_accuracies)) / len(task_accuracies)
    return avg_acc, forgetting

Common pitfalls

• The paper does not specify the exact mathematical formula for 'Forgetting'; standard continual learning literature uses either average drop or max drop, so implementations may vary and must be explicitly stated.
• The evaluation strictly uses a fixed rehearsal buffer of 50 audio-visual pairs per category from old tasks. Omitting or resizing this buffer will invalidate comparisons with the reported baselines.
• The setting is class-incremental with separate test sets per task, meaning the model must predict absolute class labels without task IDs. Confusing this with task-incremental evaluation will inflate results.

Evidence (verbatim from paper)

Following the prior work, we use the class-incremental setting of T=4 sequential classification tasks with equal sizes of categories, where each task has a separate test set. With the common metrics in previous methods, we apply Average accuracy and Forgetting for comprehensive evaluation. Higher Average accuracy is better, and lower Forgetting is better.

Citation

@misc{mo2023cign,
  title={Class-Incremental Grouping Network for Continual Audio-Visual Learning},
  author={Mo et al. (2023)},
  year={2023},
  note={arXiv:2309.05281}
}

• arXiv: 2309.05281