Perr Eval

perr-eval

Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark — Gao et al. (2021) (arXiv:2109.11243, 2021)

What this evaluates

This benchmark evaluates a model's ability to recognize the emotional relationship (e.g., intimate, hostile, neutral) between two interacting characters in drama videos. It probes multi-modal fusion capabilities by requiring the model to integrate visual, audio, and textual cues to classify pairwise interactions.

Datasets

• ERATO — total 31182; splits: train (-1), val (-1), test (-1); repo https://github.com/CTI-VISION/PERR

Metrics

• Micro-F1 (primary) — range: [0, 1]
  • Micro-averaged F1 score computed by aggregating the total true positives, false positives, and false negatives across all classes before calculating precision and recall, then deriving the harmonic mean.
• Macro-F1 — range: [0, 1]
  • Macro-averaged F1 score computed as the unweighted mean of the F1 scores calculated independently for each class, giving equal weight to all categories regardless of class imbalance.

Input / output format

Input: Multi-modal features extracted from video clips: background visual features, person-wise visual features (facial expression, posture, relative position to the other person), audio features, and textual (subtitle) features.

Output: A single categorical label representing the pairwise emotional relationship between two characters, evaluated under either a 5-class or 3-class taxonomy.

Scoring recipe

def compute_f1(y_true, y_pred, average='micro'):
    # Calculate per-class or global TP, FP, FN
    if average == 'micro':
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == p)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != p)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / len(y_true) if len(y_true) > 0 else 0
    else: # macro
        classes = set(y_true)
        f1_scores = []
        for c in classes:
            tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
            fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
            fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0
            f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
        return sum(f1_scores) / len(f1_scores)
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Ignoring paired interactive characters (person-wise features) causes a sharp performance drop (~5% Macro-F1), as the task fundamentally relies on character interaction rather than isolated cues.
• Relying on a single modality instead of fusing visual, audio, and text significantly underperforms the full multi-modal setup, as key emotional signals are distributed across modalities.
• Omitting temporal context in the fusion stage reduces Macro-F1 by approximately 1–2.75%, since emotional relationships in drama videos unfold over time.

Evidence (verbatim from paper)

The performances in terms of Micro-F1 and Macro-F1 on ERATO for coarse and fine-grained categories are shown in Table 3. Specifically, it achieves Micro-F1 of 66.55%, Macro-F1 of 49.81% for the task of 5 categories, and Micro-F1 of 70.05%, Macro-F1 of 61.73% for the task of 3 categories.

Citation

@misc{gao2021perr,
  title={Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark},
  author={Gao et al. (2021)},
  year={2021},
  note={arXiv:2109.11243}
}

• arXiv: 2109.11243