Vclimb Eval

vclimb-eval

vCLIMB: A Novel Video Class Incremental Learning Benchmark — Villa et al. (2022) (arXiv:2201.09381, 2022)

What this evaluates

This benchmark evaluates video class incremental learning capabilities, testing a model's ability to sequentially learn new action categories while retaining knowledge of previous tasks using limited episodic memory. It specifically probes how well models handle temporal consistency, frame-level memory selection, and classification on both trimmed and untrimmed video data without catastrophic forgetting.

Datasets

• UCF101 — total 13300; splits: train (-1)
• Kinetics — total 300000; splits: train (-1)
• ActivityNet-Trim — total 20000; splits: train (-1)
• ActivityNet-Untrim — total 20000; splits: train (-1)

Metrics

• Final Average Accuracy (Acc) (primary) — range: [0, 1]
  • The average classification accuracy of the model evaluated on all learned tasks, including the last task it was trained on. It measures how average performance degrades as new tasks are learned.
• Backward Forgetting (BWF) — range: [0, 1]
  • Measures the influence of a newly learned task on the performance of previous tasks. Calculated as BWF_i = 1/(N_i-1) * sum_{j=1}^{N_i-1} (R_{j,j} - R_{N_i,j}), where R_{j,j} is accuracy on task j after learning task j, and R_{N_i,j} is accuracy on task j after learning task i. Final BWF is reported at the end of all N tasks.

Input / output format

Input: Video clips represented as sequences of frames. For trimmed setups, only frames within labeled action segments are provided. For untrimmed setups, the entire video including background frames is provided, with the whole video assigned the primary action label.

Output: A single class label prediction corresponding to the action category in the video.

Scoring recipe

# R[t, i] stores accuracy on task t after learning task i
for i in range(num_tasks):
    model.train_on_task(i)
    for t in range(i + 1):
        R[t, i] = compute_accuracy(task_data[t], model)

# Final Average Accuracy
acc = sum(R[t, t] for t in range(num_tasks)) / num_tasks

# Backward Forgetting (BWF)
bwf = sum(R[j, j] - R[j, num_tasks - 1] for j in range(num_tasks - 1)) / (num_tasks - 1)

Common pitfalls

• Memory size is strictly defined by the number of stored frames, not videos, to prevent bias toward longer videos maximizing storage.
• The untrimmed setup assigns the same label to all frames in a video, including background frames, which differs from standard trimmed action recognition protocols.
• Temporal consistency regularization requires evaluating performance on both original and temporally down-sampled versions of memory videos during training to enforce representation stability.

Evidence (verbatim from paper)

Metrics for Video Continual Learning (CL). In vCLIMB, we use the standard CL metrics: Final Average Accuracy (Acc) and Backward Forgetting (BWF). Acc is the average classification accuracy of the model evaluated on all learned tasks, including the last task it was trained on [[23], [37]]. This metric is essential to show how the average performance of the model degrades as it learns new tasks. BWF complements Acc and measures the influence of the learned task $i$ in the performance of the previous tasks [[23]], as: $BWF_{i}=\frac{1}{N_{i}-1}\sum_{j=1}^{N_{i}-1}(R_{j,j}-R_{N_{i},j})$

Citation

@misc{villa2022vclimb,
  title={vCLIMB: A Novel Video Class Incremental Learning Benchmark},
  author={Villa et al. (2022)},
  year={2022},
  note={arXiv:2201.09381}
}

• arXiv: 2201.09381