qhjqhj00 ocl-accuracy-retention-eval
Online Continual Learning Without the Storage Constraint — Prabhu et al. (2023) (arXiv:2305.09253, 2023)
What this evaluates
Evaluates an online continual learning model's ability to rapidly adapt to incoming data streams while retaining knowledge of past classes without catastrophic forgetting, under strict computational budgets and fixed feature extractors.
Datasets
- CGLM — total ?; splits: pretrain (-1), train (-1), test (-1)
- CLOC — total 39000000; splits: pretrain (-1), train (-1), test (-1)
Metrics
a_t(primary) — range: [0, 1]- Average online accuracy until timestep t, computed as the cumulative fraction of correct predictions over all timesteps seen so far: a_t = (1/t) * sum_{i=1 to t} 1(y_i == y_hat_i).
IR_h— range: [0, 1]- Information retention over the last h timesteps, measuring catastrophic forgetting mitigation: IR_h = (1/h) * sum_{t=T-h to T} 1(y_t == y_hat_t).
Input / output format
Input: Sequential incoming images processed through a fixed pretrained ResNet50 backbone and a 2-layer MLP projector to 256-dimensional normalized features.
Output: Predicted class label y_hat_i for each incoming image.
Scoring recipe
def compute_a_t(predictions, labels, t):
correct = sum(1 for y, y_hat in zip(labels[:t], predictions[:t]) if y == y_hat)
return correct / t
def compute_IR_h(predictions, labels, T, h):
correct = sum(1 for y, y_hat in zip(labels[T-h:T], predictions[T-h:T]) if y == y_hat)
return correct / h
Common pitfalls
- Confusing online accuracy (a_t) with final test accuracy; a_t is computed cumulatively over all timesteps seen so far, not just the current batch.
- Assuming methods update the backbone; all baselines and ACM use a fixed pretrained ResNet50 feature extractor, with only the classifier/adaptor updated.
- Overlooking the computational budget constraint; fairness is enforced by limiting all methods to one gradient update per batch, regardless of storage availability.
Evidence (verbatim from paper)
We follow Cai et al. (2021), measuring average online accuracy until the current timestep $t$ ($a_{t}$) as a metric for measuring rapid adaptation, given by $a_{t}=\nicefrac{{1}}{{t}}\sum_{i=1}^{t}\mathds{1}{y{i}=\hat{y}{i}}$ where $\mathds{1}{(\cdot)}$ is the indicator function. We additionally measure information retention, i.e. mitigating catastrophic forgetting, after online training on unseen samples from a test set. Formally, information retention for $h$ timesteps (${IR}{h}$) at time $T$, is defined as ${IR}{h}=\nicefrac{{1}}{{h}}\sum_{t=T-h}^{T}\mathds{1}{y{t}=\hat{y}_{t}}$.
Citation
@misc{prabhu2023online,
title={Online Continual Learning Without the Storage Constraint},
author={Prabhu et al. (2023)},
year={2023},
note={arXiv:2305.09253}
}
- arXiv: 2305.09253