Dfme Eval

dfme-eval

DFME: A New Benchmark for Dynamic Facial Micro-expression Recognition — Sirui Zhao et al. (2023) (arXiv:2301.00985, 2023)

What this evaluates

Evaluates automatic dynamic facial micro-expression recognition (MER) models on a large-scale spontaneous micro-expression dataset. It probes the model's ability to classify subtle, high-frame-rate facial movements across seven emotion categories while handling class imbalance and variable video lengths.

Datasets

• DFME — total 7275; splits: train (-1), test (-1)

Metrics

• Accuracy (ACC) (primary) — range: [0, 1]
  • Average per-class accuracy: ACC = (1/K) * Σ(TP_i / N_i). (Note: Paper writes sum in Eq. 8, but results table shows values in [0,1], indicating averaging over K=7 classes).
• Unweighted F1-Score (UF1) — range: [0, 1]
  • Macro-averaged F1: UF1 = (1/K) * Σ(2TP_i / (2TP_i + FP_i + FN_i)) across K=7 classes.
• Unweighted Average Recall (UAR) — range: [0, 1]
  • Macro-averaged recall: UAR = (1/K) * Σ(TP_i / N_i) across K=7 classes.

Input / output format

Input: 16-frame video clips of cropped and aligned facial regions. Frames are sampled using a temporal adaptive strategy based on annotated onset, apex, and offset frames. Input resolution varies by backbone (e.g., 224×224 for R3D/I3D/D3D, 160×160 for P3D).

Output: Single-class classification prediction from 7 emotion categories: disgust, surprise, happiness, fear, sadness, anger, contempt.

Scoring recipe

# 10-fold cross-validation: average results over 10 runs
acc_sum, uf1_sum, uar_sum = 0, 0, 0
for fold in range(10):
    preds, gold = get_fold_data(fold)
    tp, fp, fn = compute_tp_fp_fn(preds, gold, classes=7)
    N = [count(gold == c) for c in classes]
    acc_sum += sum(tp[i] / N[i] for i in range(7)) / 7
    uf1_sum += sum(2*tp[i] / (2*tp[i] + fp[i] + fn[i]) for i in range(7)) / 7
    uar_sum += sum(tp[i] / N[i] for i in range(7)) / 7
final_acc = acc_sum / 10
final_uf1 = uf1_sum / 10
final_uar = uar_sum / 10

Common pitfalls

• Class imbalance heavily biases Accuracy (ACC) towards the majority class (disgust), making UF1 and UAR more reliable indicators of true performance.
• Temporal sampling strategy significantly impacts results; uniform sampling is too coarse for micro-expressions, so adaptive sampling around onset/apex/offset frames is required.
• Preprocessing must use only the onset frame for alignment and cropping to avoid landmark errors introduced by subtle movements in other frames.

Evidence (verbatim from paper)

In addition, three commonly used MEs classification indicators, namely Accuracy, Unweighted F1-Score and Unweighted Average Recall, were used to evaluate the MER performance. Specifically, before calculating them, we need to obtain the True Positive (TP_i), False Positive (FP_i), and False Negative (FN_i) for each class i (K classes in total, and K=7 in DFME). In the end, we took the average results of ten experiments as the final result.

Citation

@misc{zhao2023dfme,
  title={DFME: A New Benchmark for Dynamic Facial Micro-expression Recognition},
  author={Sirui Zhao et al. (2023)},
  year={2023},
  note={arXiv:2301.00985}
}

• arXiv: 2301.00985