Egoscreen Emotion Eval

egoscreen-emotion-eval

Watching Movies Like a Human: Egocentric Emotion Understanding for Embodied Companions — Ze Dong et al. (2026) (arXiv:2604.15823, 2026)

What this evaluates

Evaluates a model's ability to predict human emotional responses to movie scenes from an egocentric, first-person screen-view perspective. It probes multimodal long-context reasoning by combining visual frames, audio cues, and narrative summaries to handle domain shifts from cinematic to realistic viewing conditions.

Datasets

• EgoScreen-Emotion (ESE) — total 224; splits: train (-1), test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted emotion labels out of the total number of instances.
• Macro-F1 — range: percent
  • Macro-averaged F1 score computed across 10 emotion classes, treating each class equally regardless of its frequency in the dataset.
• Weighted-F1 — range: percent
  • Weighted-averaged F1 score computed across 10 emotion classes, weighting each class's F1 by its support (number of true instances).

Input / output format

Input: Visual input consisting of either 1 or 3 frames sampled at 5-second intervals, optionally augmented with synchronized audio cues and narrative summaries.

Output: A single predicted emotion class from a predefined 10-class taxonomy.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    acc = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    macro_f1 = macro_f1_score(gold_labels, predictions, average='macro')
    weighted_f1 = weighted_f1_score(gold_labels, predictions, average='weighted')
    return acc, macro_f1, weighted_f1

Common pitfalls

• Models trained on clean cinematic footage suffer a >40% drop in Macro-F1 when evaluated on egocentric FPV data, highlighting a severe domain shift that requires FPV-specific fine-tuning.
• The dataset has a long-tailed label distribution, so Accuracy alone can be misleading; Macro-F1 is reported to fairly evaluate minority emotion classes.
• Input format varies across ablations (1F vs 3F, with/without audio/narrative), so direct comparison requires matching the exact configuration.

Evidence (verbatim from paper)

We formulate emotion prediction as a 10-class classification task, deriving ground-truth labels via the confidence-summed aggregation detailed in Sec.3.2. To rigorously account for the long-tailed distribution, we report Accuracy, Macro-F1, and Weighted-F1.

Citation

@misc{dong2026egoscreen,
  title={Watching Movies Like a Human: Egocentric Emotion Understanding for Embodied Companions},
  author={Ze Dong et al. (2026)},
  year={2026},
  note={arXiv:2604.15823}
}

• arXiv: 2604.15823