Emotiw2018 Group Er Eval

emotiw2018-group-er-eval

Non-Volume Preserving-based Fusion to Group-Level Emotion Recognition on Crowd Videos — Quach et al. (2018) (arXiv:1811.11849, 2018)

What this evaluates

Evaluates a model's ability to recognize and aggregate facial emotions across multiple individuals in a crowd scene. It probes the system's capacity to capture complex spatial dependencies among overlapping facial expressions to predict the dominant group-level emotion.

Datasets

• EmotiW2018 — total 17172; splits: train (9815), val (4346), test (3011)
• GECV — total 627; splits: train (565), test (62)

Metrics

• mean accuracy (mAC) (primary) — range: [0, 1]
  • Average of per-class accuracies across Neutral, Positive, and Negative emotion categories.
• Unweighted Average Recall (UAR) — range: [0, 1]
  • Average of recall scores computed independently for each emotion class, treating all classes equally regardless of sample size.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall, typically reported as the macro-average across classes.

Input / output format

Input: Crowd images or video frames containing multiple faces, with ground-truth group-level emotion labels.

Output: A single predicted group-level emotion label per image/video (Neutral, Positive, or Negative).

Scoring recipe

def compute_metrics(predictions, gold):
    classes = ['Neutral', 'Positive', 'Negative']
    class_accuracies = []
    class_recalls = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        class_accuracies.append(tp / (tp + fn) if (tp + fn) > 0 else 0)
        class_recalls.append(tp / (tp + fn) if (tp + fn) > 0 else 0)
    mAC = sum(class_accuracies) / len(classes)
    UAR = sum(class_recalls) / len(classes)
    return mAC, UAR

Common pitfalls

• Fusion strategy choice (averaging vs. concatenation vs. NVPF) drastically changes performance, so baselines must use identical feature extractors.
• Class imbalance in crowd scenes can skew overall accuracy; UAR or F1 should be checked to ensure minority emotions (e.g., Negative) are not ignored.
• Low-resolution and overlapping faces in crowd videos require robust feature extraction before fusion.

Evidence (verbatim from paper)

Overall/mean accuracy, per class accuracy, mean F1, and Unweighted Average Recall (UAR) are reported in this experiment. EmotiW 2018 group-based ER sub-challenge is an extension of the sub-challenge in EmotiW 2017 with 9,815 images for training, 4,346 images for validation, and 3,011 for testing, respectively.

Citation

@misc{quach2018nvpf,
  title={Non-Volume Preserving-based Fusion to Group-Level Emotion Recognition on Crowd Videos},
  author={Quach et al. (2018)},
  year={2018},
  note={arXiv:1811.11849}
}

• arXiv: 1811.11849