# Perr Eval

> This benchmark evaluates a model's ability to recognize the emotional relationship (e.g., intimate, hostile, neutral) between two interacting characters in drama videos. It probes multi-modal fusion capabilities by requiring the model to integrate visual, audio, and textual cues to classify pairwise interactions. Use when the user wants to benchmark on ERATO, or asks about evaluating this task. Reports Micro-F1.

- Skill: `qhjqhj00/perr-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/perr-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/perr-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/perr-eval

---


# perr-eval

> Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark — Gao et al. (2021) (arXiv:2109.11243, 2021)

## What this evaluates

This benchmark evaluates a model's ability to recognize the emotional relationship (e.g., intimate, hostile, neutral) between two interacting characters in drama videos. It probes multi-modal fusion capabilities by requiring the model to integrate visual, audio, and textual cues to classify pairwise interactions.

## Datasets

- **ERATO** — total 31182; splits: train (-1), val (-1), test (-1); repo https://github.com/CTI-VISION/PERR

## Metrics

- `Micro-F1` **(primary)** — range: [0, 1]
  - Micro-averaged F1 score computed by aggregating the total true positives, false positives, and false negatives across all classes before calculating precision and recall, then deriving the harmonic mean.
- `Macro-F1` — range: [0, 1]
  - Macro-averaged F1 score computed as the unweighted mean of the F1 scores calculated independently for each class, giving equal weight to all categories regardless of class imbalance.

## Input / output format

**Input**: Multi-modal features extracted from video clips: background visual features, person-wise visual features (facial expression, posture, relative position to the other person), audio features, and textual (subtitle) features.

**Output**: A single categorical label representing the pairwise emotional relationship between two characters, evaluated under either a 5-class or 3-class taxonomy.

## Scoring recipe

```python
def compute_f1(y_true, y_pred, average='micro'):
    # Calculate per-class or global TP, FP, FN
    if average == 'micro':
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == p)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != p)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / len(y_true) if len(y_true) > 0 else 0
    else: # macro
        classes = set(y_true)
        f1_scores = []
        for c in classes:
            tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
            fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
            fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0
            f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
        return sum(f1_scores) / len(f1_scores)
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Ignoring paired interactive characters (person-wise features) causes a sharp performance drop (~5% Macro-F1), as the task fundamentally relies on character interaction rather than isolated cues.
- Relying on a single modality instead of fusing visual, audio, and text significantly underperforms the full multi-modal setup, as key emotional signals are distributed across modalities.
- Omitting temporal context in the fusion stage reduces Macro-F1 by approximately 1–2.75%, since emotional relationships in drama videos unfold over time.

## Evidence (verbatim from paper)

> The performances in terms of Micro-F1 and Macro-F1 on ERATO for coarse and fine-grained categories are shown in Table 3. Specifically, it achieves Micro-F1 of 66.55%, Macro-F1 of 49.81% for the task of 5 categories, and Micro-F1 of 70.05%, Macro-F1 of 61.73% for the task of 3 categories.

## Citation

```bibtex
@misc{gao2021perr,
  title={Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark},
  author={Gao et al. (2021)},
  year={2021},
  note={arXiv:2109.11243}
}
```

- arXiv: 2109.11243