Face Mtl Eval

face-mtl-eval

Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study — Kollias et al. (2021) (arXiv:2105.03790, 2021)

What this evaluates

Evaluates a multi-task learning framework for face analysis across heterogeneous tasks including valence-arousal estimation, action unit detection, expression classification, and face recognition. It probes the model's ability to jointly learn from diverse, in-the-wild and lab-controlled facial datasets while mitigating negative transfer through distribution matching.

Datasets

• Aff-Wild — total 1250000; splits: train (1000000), test (216000)
• AffectNet — total 1000000; splits: train (321000), val (5000)
• RAF-DB — total 15200; splits: train (12200), test (3000)
• DISFA — total 130515; splits: train (-1), test (-1)
• GFT — total 130000; splits: train (108000), test (24500)
• BP4D — total 223000; splits: train (75600), val (71200), test (75700)
• CelebA — total 202600; splits: train (160000), val (20000), test (20000)

Metrics

• CCC (primary) — range: [-1, 1]
  • Concordance Correlation Coefficient measuring the agreement between predicted and ground-truth continuous values (valence/arousal). It combines precision and accuracy of the prediction relative to the line of identity.
• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, computed per action unit or emotion category and averaged across classes.
• Mean diagonal of confusion matrix — range: [0, 1]
  • Sum of correctly classified samples divided by total samples, equivalent to overall classification accuracy.
• AFA Score — range: [0, 1]
  • Average of the F1 score and accuracy across all action units or emotion categories.

Input / output format

Input: Cropped and aligned facial images resized to 112x112x3 pixels with intensity values normalized to [-1, 1].

Output: Continuous predictions for valence and arousal (range [-1, 1]), binary/multi-class predictions for action units and expressions, and attribute/identity labels.

Scoring recipe

def compute_metrics(pred, gold, task_type):
    if task_type == 'continuous':
        mu_p, mu_g = np.mean(pred), np.mean(gold)
        var_p, var_g = np.var(pred), np.var(gold)
        cov = np.cov(pred, gold)[0, 1]
        return 2 * cov / (var_p + var_g + (mu_p - mu_g)**2)
    elif task_type == 'classification':
        cm = confusion_matrix(gold, pred)
        diag = np.trace(cm) / cm.sum()
        f1 = f1_score(gold, pred, average='macro')
        return {'accuracy': diag, 'f1': f1, 'afa': (diag + f1) / 2}

Common pitfalls

• AffectNet, BP4D, and BP4D+ do not release official test sets; the protocol uses the validation set for testing, which may inflate generalization estimates.
• Video datasets like Aff-Wild contain temporally correlated frames, but the CNN processes frames independently; median filtering of frame-level predictions is required to match challenge protocols.
• AffectNet has overlapping/inconsistent annotations between valence-arousal and discrete expressions; strict data cleaning (e.g., radius checks for neutral/sad) is mandatory before training.

Evidence (verbatim from paper)

We use: i) the CCC for Aff-Wild and Aff-Wild2 (CCC was the evaluation criterion of the respective Challenges), Affectnet and AFEW-VA, ii) the mean diagonal value of the confusion matrix for RAF-DB (this criterion was selected for evaluating the performance on this database by [[34]]); the accuracy for AffectNet, iii) the F1 score for DISFA, GFT, BP4D and BP4D+ (this metric was the evaluation criterion of the FERA 2015 and 2017 Challenges); for AU detection in EmotioNet the Challenge’s metric was the average between: a) the mean (across all AUs) F1 score and b) the mean (across all AUs) accuracy; for the expression classification, it was the average between: a) the average (across all emotions) F1 score and b) the unweighted average recall (UAR) over all emotion categories, iv) the total accuracy and average F1 score for the attributes and ids in CelebA.

Citation

@misc{kollias2021distribution,
  title={Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study},
  author={Kollias et al. (2021)},
  year={2021},
  note={arXiv:2105.03790}
}

• arXiv: 2105.03790