qhjqhj00 clif-eval
Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning — Jin et al. (2021) (arXiv:2104.08808, 2021)
What this evaluates
Evaluates a model's ability to accumulate knowledge across a sequence of NLP tasks (continual learning) while maintaining performance on previously seen tasks and generalizing to new few-shot tasks.
Datasets
- CLIF-26 — total ?; splits: train (-1), test (-1); repo https://github.com/INK-USC/CLIF
- CLIF-55 — total ?; splits: train (-1), test (-1); repo https://github.com/INK-USC/CLIF
Metrics
Final Accuracy(primary) — range: percent- Standard classification accuracy computed over all upstream tasks after continual learning is complete.
Instant Accuracy— range: percent- Accuracy measured on upstream tasks immediately after learning each task, without further fine-tuning.
Few-shot Accuracy— range: percent- Accuracy measured on unseen tasks using only K labeled examples per task for adaptation.
Input / output format
Input: Text pairs (input, label) formatted for a text-to-text model (BART), with task representations computed as average encoder-decoder latent activations over training examples.
Output: Predicted text labels or class tokens generated by the model.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
def compute_relative_improvement(metric_val, baseline_val):
return ((metric_val - baseline_val) / baseline_val) * 100
Common pitfalls
- Confusing Instant Accuracy (performance on upstream tasks during/after continual learning) with Few-shot Accuracy (performance on unseen tasks with limited examples).
- Failing to use the correct zero-knowledge baseline (BART-Adapter-Single or BiHNet-Single) when computing relative improvements (Δ_Inst. and Δ_FS.).
Evidence (verbatim from paper)
Final accuracy (Final Acc.) and instant accuracy (Instant Acc.) over upstream tasks and accuracy over few-shot learning tasks (Few-shot Acc.) on CLIF-26 and CLIF-55 tasks. We compute relative improvement of instant accuracy ($\Delta_{\text{Inst.}}$) and few-shot accuracy ($\Delta_{\text{FS}}$) over zero-knowledge baselines (the better one between BART-Adapter-Single and BiHNet-Single for BiHNet, and BART-Single for BART approaches).
Citation
@misc{jin2021learn,
title={Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning},
author={Jin et al. (2021)},
year={2021},
note={arXiv:2104.08808}
}
- arXiv: 2104.08808