Facebehaviornet Eval

facebehaviornet-eval

Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network — Kollias et al. (2019) (arXiv:1910.11111, 2019)

What this evaluates

Evaluates a multi-task facial analysis model's ability to jointly predict continuous affect (valence/arousal), discrete facial expressions, and facial action units from in-the-wild and lab-controlled face images. It probes the model's capacity for task-coupled learning and generalization across heterogeneous annotation schemes and domains.

Datasets

• Aff-Wild — total 1250000; splits: train (-1), val (-1)
• AffectNet — total 1000000; splits: train (-1), val (-1)
• AFEW — total 1809; splits: val (-1)
• RAF-DB — total 15200; splits: val (-1)
• EmotioNet — total 1000000; splits: val (-1)
• DISFA — total 260000; splits: val (-1)
• BP4D — total 223000; splits: val (-1)
• BP4D+ — total ?; splits: val (-1)

Metrics

• CCC (primary) — range: [-1, 1]
  • Concordance Correlation Coefficient. Measures precision and accuracy of the predicted mean relative to the true mean: CCC = 2rhosigma_x*sigma_y / (sigma_x^2 + sigma_y^2 + (mu_x - mu_y)^2).
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall. Used for AU detection and expression classification on most datasets.
• Total accuracy — range: [0, 1]
  • Percentage of correctly classified samples. Used for AFEW.
• Mean diagonal of confusion matrix — range: [0, 1]
  • Average of correct classification rates per class. Used for RAF-DB.
• UAR — range: [0, 1]
  • Unweighted Average Recall. Average of recall scores across all classes. Used for EmotioNet expression classification.

Input / output format

Input: 96x96x3 RGB facial images with intensity normalized to [-1, 1], extracted using SSH detector and aligned via 5 facial landmarks.

Output: Continuous values for valence and arousal; categorical labels for basic/compound expressions; binary/intensity scores for Action Units (AUs).

Scoring recipe

import numpy as np
from sklearn.metrics import f1_score, accuracy_score

def compute_ccc(y_true, y_pred):
    mean_true, mean_pred = np.mean(y_true), np.mean(y_pred)
    var_true, var_pred = np.var(y_true), np.var(y_pred)
    cov = np.cov(y_true, y_pred)[0, 1]
    return (2 * cov) / (var_true + var_pred + (mean_true - mean_pred)**2)

def score_dataset(y_true, y_pred, metric_type):
    if metric_type == 'CCC':
        return compute_ccc(y_true, y_pred)
    elif metric_type == 'F1':
        return f1_score(y_true, y_pred, average='macro')
    elif metric_type == 'Accuracy':
        return accuracy_score(y_true, y_pred)
    elif metric_type == 'UAR':
        recalls = [np.mean(y_pred[y_true==c] == c) for c in np.unique(y_true)]
        return np.mean(recalls)
    return None

Common pitfalls

• Datasets like AffectNet, AFEW, BP4D, and BP4D+ lack official test splits; the authors repurpose the validation set for testing and split the training set 85/15.
• Each dataset uses a different primary metric (CCC for VA, F1 for AUs, accuracy for AFEW, mean diagonal for RAF-DB, composite scores for EmotioNet).
• EmotioNet AU evaluation averages mean F1 and mean accuracy across all AUs, which differs from standard macro-F1.

Evidence (verbatim from paper)

We use: i) the CCC for Aff-Wild (CCC was the evaluation criterion of Aff-Wild Challenge) and Affectnet, ii) the total accuracy for AFEW (this metric was the evaluation criterion of the EmotiW Challenges), the mean diagonal value of the confusion matrix for RAF-DB (this criterion was selected for evaluating the performance on this database by [li2017reliable]), the F1 score for AffectNet, iii) the F1 score for DISFA, BP4D and BP4D+ (this metric was the evaluation criterion of the FERA 2015 and 2017 Challenges); for AU detection in EmotioNet the Challenge’s metric was the average between: a) the mean (across all AUs) F1 score and b) the mean (across all AUs) accuracy; for the expression classification, it was the average between: a) the mean (across all emotions) F1 score and b) the unweighted average recall (UAR) over all emotion categories.

Citation

@misc{kollias2019facebehavior,
  title={Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network},
  author={Kollias et al. (2019)},
  year={2019},
  note={arXiv:1910.11111}
}

• arXiv: 1910.11111