Open Domain Continual Learning Eval

open-domain-continual-learning-eval

CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning — Li et al. (2024) (arXiv:2403.10245, 2024)

What this evaluates

Evaluates open-domain continual learning (ODCL) in vision-language models by measuring how well a model adapts to a stream of new image classification domains while preserving previously learned knowledge and zero-shot capabilities on unseen domains.

Datasets

• Aircraft — total ?; splits: test (-1)
• Caltech101 — total ?; splits: test (-1)
• CIFAR100 — total ?; splits: test (-1)
• DTD — total ?; splits: test (-1)
• EuroSAT — total ?; splits: test (-1)
• Flowers — total ?; splits: test (-1)
• Food — total ?; splits: test (-1)
• MNIST — total ?; splits: test (-1)
• OxfordPet — total ?; splits: test (-1)
• StanfordCars — total ?; splits: test (-1)
• SUN397 — total ?; splits: test (-1)

Metrics

• Avg (primary) — range: percent
  • Average accuracy over all datasets across all time steps. Avg = (1/T) * sum_{t=1}^{T} A_t, where A_t = (1/T) * sum_{i=1}^{T} A_t^i and A_t^i is accuracy on dataset t after training on task i.
• Last — range: percent
  • Average accuracy across all domain datasets at the final time step (after training on all T tasks).
• Transfer — range: percent
  • Average zero-shot accuracy on unseen domains. Transfer = (1/T) * sum_{t=1}^{T} T_t, where T_t = (1/(t-1)) * sum_{i=1}^{t-1} A_t^i, measuring accuracy on dataset t before it was seen.
• Forgetting — range: percent
  • Average accuracy over all datasets from the learned time step to the last time step. Forgetting = (1/T) * sum_{t=1}^{T} F_t, where F_t = (1/(T-t+1)) * sum_{i=t}^{T} A_t^i.

Input / output format

Input: Image input processed by a frozen CLIP image encoder with task-specific prompts; text class vocabulary processed by a frozen CLIP text encoder with LoRA-based PEFT modules.

Output: Predicted class labels for each image, evaluated as classification accuracy (%) on seen and unseen domains.

Scoring recipe

T = total tasks
A = dict mapping (dataset, time_step) -> accuracy

# Last: accuracy at final step
last_acc = [A[(d, T)] for d in datasets]
Last = mean(last_acc)

# Avg: average accuracy across all steps and datasets
avg_acc = [mean([A[(d, i)] for i in range(1, T+1)]) for d in datasets]
Avg = mean(avg_acc)

# Transfer: zero-shot accuracy on unseen domains
transfer_acc = []
for t in range(1, T+1):
    unseen_accs = [A[(d, i)] for i in range(1, t)]
    transfer_acc.append(mean(unseen_accs) if unseen_accs else 0)
Transfer = mean(transfer_acc)

# Forgetting: accuracy retention from learning step to end
forget_acc = []
for t in range(1, T+1):
    retention_accs = [A[(d, i)] for i in range(t, T+1)]
    forget_acc.append(mean(retention_accs))
Forgetting = mean(forget_acc)

Common pitfalls

• ODCL-CIL (no task ID at inference) is significantly more challenging than ODCL-TIL (task ID known); results drop noticeably in CIL settings.
• Avg and Transfer metrics are only applicable to methods with zero-shot capabilities (e.g., CLIP, ZSCL, CoLeCLIP); closed-domain baselines like CODA-Prompt and LAE lack these metrics and should not be compared on them.
• The paper primarily reports results under Order-I (alphabetical dataset arrival); Order-II (random arrival) results are in the appendix and may yield different performance rankings.

Evidence (verbatim from paper)

To effectively evaluate the performance of open-domain CL, following [43], three performance metrics are used, namely Avg, Last, and Transfer, to evaluate the model's ability to preserve knowledge from the pre-training and downstream tasks. Additionally, we introduce a measure called Forgetting, focusing exclusively on the prevention of catastrophic forgetting of knowledge learned from the seen domains.

Citation

@misc{li2024coleclip,
  title={CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2403.10245}
}

• arXiv: 2403.10245