# Fedmosaic Eval

> Evaluates personalized federated learning (PFL) methods on heterogeneous multi-modal and text-only clients. It probes a model's ability to personalize to its own data distribution ('Self') while maintaining generalization to unseen or other clients' tasks ('Others') under both static and dynamic distribution shifts. Use when the user wants to benchmark on DRAKE, HFLB, Fed-Scope, Fed-Aya, Fed-LLM-Large, or asks about evaluating this task. Reports A_last.

- Skill: `qhjqhj00/fedmosaic-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/fedmosaic-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/fedmosaic-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/fedmosaic-eval

---


# fedmosaic-eval

> Not All Clients Are Equal: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients — Seo et al. (2025) (arXiv:2506.11024, 2025)

## What this evaluates

Evaluates personalized federated learning (PFL) methods on heterogeneous multi-modal and text-only clients. It probes a model's ability to personalize to its own data distribution ('Self') while maintaining generalization to unseen or other clients' tasks ('Others') under both static and dynamic distribution shifts.

## Datasets

- **DRAKE** — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)
- **HFLB** — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)
- **Fed-Scope** — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)
- **Fed-Aya** — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)
- **Fed-LLM-Large** — total ?; splits: PFL-Static (-1), PFL-Dynamic (-1)

## Metrics

- `A_last` **(primary)** — range: [0, 1]
  - Accuracy of the model on the evaluation set at the final training step.
- `A_AUC` — range: [0, 1]
  - Area under the accuracy curve, computed by measuring accuracy at each of the 5 evaluation intervals during training.

## Input / output format

**Input**: Multi-modal (image + text) or text-only prompts partitioned across clients. Clients use heterogeneous model architectures (e.g., LLaVA-Llama3 vs. LLaVA-Qwen variants of varying sizes).

**Output**: Accuracy scores reported as mean ± standard deviation over three random seeds. Evaluated separately for 'Self' (own client's data) and 'Others' (other clients' data).

## Scoring recipe

```python
def compute_metrics(predictions, gold, accuracy_history):
    a_last = accuracy(predictions, gold)
    a_auc = mean(accuracy_at_step(step, gold) for step in accuracy_history)
    return a_last, a_auc
# Final report: mean(a_last, a_auc) across 3 random seeds ± std
```

## Common pitfalls

- 'Self' and 'Others' refer to evaluation on the client's own data vs. other clients' data, not standard train/test splits.
- A_AUC requires tracking accuracy at 5 specific evaluation intervals during training, not just the final checkpoint.
- Baselines must be compared under identical model heterogeneity setups (e.g., LLaVA-Llama3-1B vs 3B) to avoid unfair advantages.

## Evidence (verbatim from paper)

> Metrics. We report $A_{last}$, the accuracy at the end of training, and $A_{	ext{AUC}}*(Koh et al., [2022])* , which computes the area under the accuracy curve by measuring accuracy at each evaluation period to capture intermediate performance. All experiments use five rounds of evaluation intervals and are averaged over three different random seeds, with standard deviations reported.

## Citation

```bibtex
@misc{seo2025fedmosaic,
  title={Not All Clients Are Equal: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients},
  author={Seo et al. (2025)},
  year={2025},
  note={arXiv:2506.11024}
}
```

- arXiv: 2506.11024