Adamerging Eval

adamerging-eval

AdaMerging: Adaptive Model Merging for Multi-Task Learning — Yang et al. (2023) (arXiv:2310.02575, 2023)

What this evaluates

Evaluates multi-task model merging methods on image classification tasks by measuring average accuracy across multiple datasets, generalization to unseen tasks, and robustness to image corruptions.

Datasets

• Image Classification Bench (SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, DTD) — total ?; splits: test (-1)

Metrics

• Avg Acc (primary) — range: percent
  • Average accuracy across all evaluated tasks: (1/N) * sum(accuracy_i) for i in tasks.

Input / output format

Input: Pre-trained ViT models fine-tuned on individual tasks (task vectors) and unlabeled test images for coefficient optimization; labeled test images for final evaluation.

Output: Merged model parameters (task vectors weighted by learned coefficients); class predictions for each test image.

Scoring recipe

def compute_avg_acc(predictions, gold_labels, tasks):
    accs = []
    for task in tasks:
        correct = sum(1 for p, g in zip(predictions[task], gold_labels[task]) if p == g)
        accs.append(correct / len(gold_labels[task]))
    return sum(accs) / len(accs)

Common pitfalls

• The method learns merging coefficients unsupervised on unlabeled test data, so the test set is used during the merging/optimization phase, not just for final evaluation.
• Generalization is tested on completely unseen tasks (e.g., MNIST/EuroSAT or RESISC45/SVHN) that were not included in the merging set.
• Robustness evaluation uses 7 specific image corruptions (Motion Blur, Impulse Noise, etc.) following the ImageNet-C protocol.

Evidence (verbatim from paper)

We report the average accuracy (i.e., Avg Acc) of MTL model on the test set of all tasks as an evaluation metric. ... Following Hendrycks & Dietterich (2019), we created 7 corruption test data, and examples of corrupted images are shown Fig.[6] in Appendix[B]. The results on ViT-B/32 are shown in Tab.[4].

Citation

@misc{yang2023adamerging,
  title={AdaMerging: Adaptive Model Merging for Multi-Task Learning},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2310.02575}
}

• arXiv: 2310.02575