C2prompt Eval

c2prompt-eval

C${}^2$Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning — Xu et al. (2025) (arXiv:2509.19674, 2025)

What this evaluates

Evaluates federated continual learning methods on image classification benchmarks. It probes long-term knowledge accumulation, progressive performance across sequential tasks, and the model's ability to retain old knowledge while learning new ones under non-IID client distributions.

Datasets

• ImageNet-R — total ?; splits: sequential_tasks (-1)
• DomainNet — total ?; splits: sequential_tasks (-1)
• CIFAR-100 — total ?; splits: sequential_tasks (-1)

Metrics

• Avg (primary) — range: percent
  • Mean accuracy across all seen tasks at the final training round. Indicates long-term knowledge accumulation.
• AIA — range: percent
  • Average Incremental Accuracy. Computed as the mean of (accuracy on task i at final round minus accuracy on task i at the end of task i). Indicates progressive performance.
• FM — range: percent
  • Forgetting Measure. Computed as the mean of (accuracy on task i at the end of task i minus accuracy on task i at the final round). Negative values indicate anti-forgetting.
• FT — range: percent
  • Forward Transfer. Average accuracy gain on new tasks compared to performance at the end of the previous task.
• BT — range: percent
  • Backward Transfer. Average accuracy change on old tasks after learning new tasks.
• CT — range: percent
  • Combined Transfer. Sum of FT and BT, measuring overall temporal transfer capacity.

Input / output format

Input: 224×224 resized images processed by a ViT-B/16 backbone with learnable discriminative and compensation prompts.

Output: Class predictions per image; aggregate accuracy scores reported per task and across all tasks.

Scoring recipe

def compute_fcl_metrics(task_accuracies):
    # task_accuracies: dict mapping (task_id, eval_round) -> accuracy
    final_round = max_round
    avg = np.mean([acc for t, acc in task_accuracies.items() if t[1] == final_round])
    aia = np.mean([acc_final - acc_at_task for t, acc_final, acc_at_task in incremental_accs])
    fm = np.mean([acc_at_task - acc_final for t, acc_final, acc_at_task in incremental_accs])
    ft = np.mean([new_task_acc - prev_task_acc])
    bt = np.mean([old_task_acc - prev_task_acc])
    ct = ft + bt
    return avg, aia, fm, ft, bt, ct

Common pitfalls

• Metrics are computed sequentially over task rounds rather than standard static train/val/test splits.
• Negative values for FM, FT, or BT indicate positive transfer or anti-forgetting, which contradicts standard accuracy intuition.
• Non-IID data across clients requires careful prompt aggregation to avoid semantic drift and invalid prompt fusion.

Evidence (verbatim from paper)

To evaluate the effectiveness of different FCL methods, 6 metrics are adopted in this paper, including Average Accuracy (Avg), Average Incremental Accuracy (AIA), Forgetting Measure (FM), Forward Transfer (FT), Backward Transfer (BT), Combined Transfer (CT). The configurations of the benchmarks and the details of the metrics are presented in Appendix [C].

Citation

@misc{xu2025c2prompt,
  title={C${}^2$Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2509.19674}
}

• arXiv: 2509.19674