Moviegraphs Emotion Eval

moviegraphs-emotion-eval

How you feelin'? Learning Emotions and Mental States in Movie Scenes — Srivastava et al. (2023) (arXiv:2304.05634, 2023)

What this evaluates

Predicts multi-label emotions and mental states for movie scenes and individual characters using multimodal inputs (video, dialog, character appearance). It probes long-form video understanding and the ability to integrate visual and linguistic cues for affect recognition.

Datasets

• MovieGraphs — total 7637; splits: train (-1), val (-1), test (-1)

Metrics

• mAP (primary) — range: percent
  • Mean Average Precision across K binary classification problems. AP is computed per label by sorting predictions by confidence and calculating precision-recall area, then mAP averages these APs. Note that AP depends on label frequency.

Input / output format

Input: Multimodal sequence: video frames sampled at 3 fps (up to 300 tokens), dialog utterances encoded via RoBERTa, and character features (face/person detections, tracks, clusters) with time embeddings.

Output: K binary predictions (one per emotion/mental state label) for each scene and each character in the scene.

Scoring recipe

def compute_mAP(preds, gold, K):
    aps = []
    for k in range(K):
        scores = preds[:, k]
        labels = gold[:, k]
        order = np.argsort(-scores)
        sorted_labels = labels[order]
        tp, fp = 0, 0
        precisions = []
        for y in sorted_labels:
            if y == 1: tp += 1
            else: fp += 1
            precisions.append(tp / (tp + fp))
        ap = np.mean(precisions) if tp > 0 else 0.0
        aps.append(ap)
    return np.mean(aps) * 100

Common pitfalls

• The test set is significantly harder than the validation set, causing consistent performance drops across all models.
• Average Precision (AP) is sensitive to label frequency, so performance varies considerably between frequent and rare emotions.
• Character-level prediction is inherently more challenging than scene-level prediction due to occlusion and limited visibility.

Evidence (verbatim from paper)

We use the original splits from MovieGraphs. As we have $K$ binary classification problems, we adopt mean Average Precision (mAP) to measure model performance (similar to Atomic Visual Actions [25]). Note that AP also depends on the label frequency.

Citation

@misc{srivastava2023howyoufeelin,
  title={How you feelin'? Learning Emotions and Mental States in Movie Scenes},
  author={Srivastava et al. (2023)},
  year={2023},
  note={arXiv:2304.05634}
}

• arXiv: 2304.05634