qhjqhj00 fedmosaic-eval
Not All Clients Are Equal: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients — Seo et al. (2025) (arXiv:2506.11024, 2025)
What this evaluates
Evaluates personalized federated learning (PFL) methods on heterogeneous multi-modal and text-only clients. It probes a model's ability to personalize to its own data distribution ('Self') while maintaining generalization to unseen or other clients' tasks ('Others') under both static and dynamic distribution shifts.
Datasets
- DRAKE — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)
- HFLB — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)
- Fed-Scope — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)
- Fed-Aya — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)
- Fed-LLM-Large — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)
Metrics
A_last(primary) — range: [0, 1]- Accuracy of the model on the evaluation set at the final training step.
A_AUC— range: [0, 1]- Area under the accuracy curve, computed by measuring accuracy at each of the 5 evaluation intervals during training.
Input / output format
Input: Multi-modal (image + text) or text-only prompts partitioned across clients. Clients use heterogeneous model architectures (e.g., LLaVA-Llama3 vs. LLaVA-Qwen variants of varying sizes).
Output: Accuracy scores reported as mean ± standard deviation over three random seeds. Evaluated separately for 'Self' (own client's data) and 'Others' (other clients' data).
Scoring recipe
def compute_metrics(predictions, gold, accuracy_history):
a_last = accuracy(predictions, gold)
a_auc = mean(accuracy_at_step(step, gold) for step in accuracy_history)
return a_last, a_auc
# Final report: mean(a_last, a_auc) across 3 random seeds ± std
Common pitfalls
- 'Self' and 'Others' refer to evaluation on the client's own data vs. other clients' data, not standard train/test splits.
- A_AUC requires tracking accuracy at 5 specific evaluation intervals during training, not just the final checkpoint.
- Baselines must be compared under identical model heterogeneity setups (e.g., LLaVA-Llama3-1B vs 3B) to avoid unfair advantages.
Evidence (verbatim from paper)
Metrics. We report $A_{last}$, the accuracy at the end of training, and $A_{ ext{AUC}}(Koh et al., [2022]) , which computes the area under the accuracy curve by measuring accuracy at each evaluation period to capture intermediate performance. All experiments use five rounds of evaluation intervals and are averaged over three different random seeds, with standard deviations reported.
Citation
@misc{seo2025fedmosaic,
title={Not All Clients Are Equal: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients},
author={Seo et al. (2025)},
year={2025},
note={arXiv:2506.11024}
}
- arXiv: 2506.11024