Facial Emotion Analysis Eval

facial-emotion-analysis-eval

Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis — Jiulong Wu et al. (2025) (arXiv:2511.10254, 2025)

What this evaluates

Evaluates vision-language models on facial emotion analysis tasks, including fine-grained action unit detection, categorical emotion recognition, and grounded natural language reasoning over facial expressions. The protocol tests both recognition accuracy and the model's ability to generate interpretable, AU-grounded explanations.

Datasets

• DISFA — total 87192; splits: train (52392), test (27654)
• BP4D — total 146847; splits: train (100813), test (46034)
• RAF-AU — total 4601; splits: train (3479), test (853)
• FER2013 — total 35887; splits: train (28709), test (3589)
• AffectNet — total 303330; splits: train (287618), test (3493)
• RAF-DB — total 29672; splits: train (12271), test (3068)
• FABA-Instruct — total 14379; splits: train (6060), test (314)
• FEA-20K — total 19425; splits: train (17737), test (1688); repo https://github.com/RobitsG/Facial-R1

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall across all action unit classes, typically averaged macro to handle class imbalance.
• Accuracy (primary) — range: [0, 1]
  • Ratio of correctly predicted emotion categories to total test samples.
• SEGE — range: [0, 1]
  • Aggregates AU recognition F1 score and ROUGE-L score of generated textual descriptions to evaluate joint recognition and reasoning quality.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between generated reasoning text and ground-truth reference text.
• GPT-4o-mini Evaluator — range: [0, 10]
  • Automated LLM-as-a-judge scoring semantic similarity between generated and ground-truth reasonings on a 0–10 scale.

Input / output format

Input: Single facial image (grayscale or color). For reasoning tasks, the image is accompanied by a text prompt requesting emotion/AU analysis or explanation.

Output: Predicted action unit labels (binary/multi-label), emotion category labels (multi-class), or natural language text describing the emotion and reasoning.

Scoring recipe

def compute_metrics(predictions, golds, task, dataset):
    if task == 'AU_recognition':
        return f1_score(golds, predictions, average='macro')
    elif task == 'emotion_recognition':
        return accuracy_score(golds, predictions)
    elif task == 'emotion_reasoning':
        rouge_l = rouge_l_score(golds['text'], predictions['text'])
        if dataset == 'FABA-Instruct':
            au_f1 = f1_score(golds['au'], predictions['au'], average='macro')
            return 0.5 * au_f1 + 0.5 * rouge_l
        elif dataset == 'FEA-20K':
            return rouge_l
        elif dataset == 'FEA-20K_reliability':
            return gpt4o_mini_evaluate(golds['text'], predictions['text'], scale=10)

Common pitfalls

• AU recognition is a multi-label binary classification task; using standard multi-class accuracy will severely underreport performance.
• Official splits for DISFA and BP4D are frame-level; inconsistent frame extraction or sampling changes dataset size and skews results.
• SEGE metric combines AU F1 and ROUGE-L; failing to normalize or weight them correctly breaks comparability with prior work.
• GPT-4o-mini evaluator scores are non-deterministic and sensitive to prompt phrasing, requiring fixed temperature and system prompts for reproducibility.

Evidence (verbatim from paper)

For AU recognition, we employ the F1 score*(mavadati2013disfa; zhang2014bp4d), ensuring a balanced evaluation of precision and recall. Regarding emotion recognition, we adopt accuracy (Acc.) as the primary metric, following established protocols(li2017RAF-DB). For AU-based emotion reasoning, we employ task-specific metrics. Specifically, on the FABA-Instruct dataset, we utilize the SEGE metric proposed in FABA(li2024faba), which aggregates the AU recognition F1 and the ROUGE-L score of textual descriptions. For the FEA-20K dataset, we report ROUGE-L to measure the textual similarity between generated and ground-truth reasonings. Additionally, to comprehensively evaluate the reliability of emotion reasoning, we propose employing GPT-4o-mini(hurst2024gpt4o)* as an automated evaluator for measuring semantic similarity between generated and ground-truth reasonings, with scores ranging from 0 to 10.

Citation

@misc{wu2025facialr1,
  title={Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis},
  author={Jiulong Wu et al. (2025)},
  year={2025},
  note={arXiv:2511.10254}
}

• arXiv: 2511.10254