Fscil Eval

fscil-eval

Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning — Yang et al. (2023) (arXiv:2302.03004, 2023)

What this evaluates

Probes a model's ability to learn new classes incrementally in a few-shot setting while retaining knowledge of previously learned classes, measuring resistance to catastrophic forgetting.

Datasets

• miniImageNet — total ?; splits: train (-1), val (-1), test (-1)
• CIFAR-100 — total ?; splits: train (-1), test (-1)
• CUB-200 — total ?; splits: train (-1), test (-1)

Metrics

• Average accuracy (primary) — range: percent
  • The arithmetic mean of the classification accuracy across all incremental sessions (base + K new sessions).
• Final session accuracy — range: percent
  • The classification accuracy evaluated on the test set of the last incremental session.
• Performance drop — range: percent
  • The difference in accuracy between the first session and the last session (First session Acc. - Last session Acc.), measuring catastrophic forgetting.

Input / output format

Input: Image samples with class labels, presented sequentially in incremental sessions. Each session provides a few-shot support set and a query set for evaluation.

Output: Predicted class label for each query image.

Scoring recipe

def compute_metrics(session_accuracies):
    avg_acc = sum(session_accuracies) / len(session_accuracies)
    final_acc = session_accuracies[-1]
    perf_drop = session_accuracies[0] - session_accuracies[-1]
    return {'Average accuracy': avg_acc, 'Final session accuracy': final_acc, 'Performance drop': perf_drop}

Common pitfalls

• FSCIL evaluation requires testing on all classes seen so far in each session, not just the newly introduced ones.
• Average accuracy must be computed across all sessions, not just the final one, to properly measure forgetting.
• Few-shot support set size (e.g., 1-shot vs 5-shot) must be strictly controlled, as it heavily impacts accuracy.

Evidence (verbatim from paper)

“Average Acc.” is the average accuracy of all sessions. “Final Improv.” calculates the improvement of our method in the last session.

Citation

@misc{yang2023neural,
  title={Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2302.03004}
}

• arXiv: 2302.03004