Forkmerge Eval

forkmerge-eval

ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning — Jiang et al. (2023) (arXiv:2301.12618, 2023)

What this evaluates

Evaluates the ability of auxiliary-task learning methods to mitigate negative transfer and improve target task performance across multi-task, multi-domain, and semi-supervised learning settings. It probes how well a model can dynamically combine or select auxiliary tasks without degrading the primary task.

Datasets

• NYUv2 — total 899; splits: train (636), val (159), test (654)
• DomainNet — total ?; splits: train (-1), val (-1), test (-1)
• AliExpress — total ?; splits: train (-1)
• CIFAR-10 — total ?; splits: test (-1)
• SVHN — total ?; splits: test (-1)

Metrics

• Δm (primary) — range: percent
  • Average per-task performance improvement across all evaluated tasks. Computed by averaging the relative or absolute improvement of each method over the Single Task Learning (STL) baseline. Exact normalization formula is defined in Appendix C.1.

Input / output format

Input: RGB images for vision tasks (NYUv2, DomainNet, CIFAR-10, SVHN); tabular/sequence features for recommendation tasks (AliExpress).

Output: Task-specific predictions: segmentation masks, depth/normal maps, class labels, or click-through rate probabilities.

Scoring recipe

improvements = []
for task in tasks:
    baseline = evaluate(model_STL, dataset[task])
    method = evaluate(model_method, dataset[task])
    if metric_is_error(task):
        improvement = (baseline - method) / baseline
    else:
        improvement = (method - baseline) / baseline
    improvements.append(improvement)
delta_m = sum(improvements) / len(improvements) * 100

Common pitfalls

• Negative transfer is algorithm-dependent, not just task-dependent; results vary significantly across different ATL baselines.
• DomainNet lacks a provided validation set, requiring a random 50% test split for hyperparameter tuning, which may introduce variance.
• Aggregating diverse metrics (accuracy, IoU, error rates) into Δm requires careful normalization to avoid scale bias across tasks.

Evidence (verbatim from paper)

Since different tasks have varying evaluation metrics, we will report the average per-task performance improvement for each method using Δm , as defined in Appendix C.1.

Citation

@misc{jiang2023forkmerge,
  title={ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning},
  author={Jiang et al. (2023)},
  year={2023},
  note={arXiv:2301.12618}
}

• arXiv: 2301.12618