Vad Prediction Eval

vad-prediction-eval

Unveiling Affective Polarization Trends in Parliamentary Proceedings — Goldin et al. (2025) (arXiv:2512.05231, 2025)

What this evaluates

Evaluates a model's ability to predict continuous emotional dimensions (Valence, Arousal, Dominance) from text. Specifically probes the model's capacity to capture affective polarization signals in parliamentary discourse.

Datasets

• Knesset VAD Annotation — total 120; splits: test (120); repo https://github.com/HaifaCLG/Polarization

Metrics

• Pearson correlation (primary) — range: [-1, 1]
  • Computes the Pearson correlation coefficient between the model's predicted continuous VAD scores and the manually annotated gold scores for each dimension (Valence, Arousal, Dominance) separately.

Input / output format

Input: Hebrew sentence text.

Output: Three continuous floating-point scores in the [0, 1] range representing Valence, Arousal, and Dominance.

Scoring recipe

def compute_vad_metrics(predictions, gold):
    # predictions and gold are arrays of shape (N, 3) for V, A, D
    metrics = {}
    for i, dim in enumerate(['Valence', 'Arousal', 'Dominance']):
        metrics[dim] = pearsonr(predictions[:, i], gold[:, i]).statistic
    return metrics

Common pitfalls

• VAD scores are derived from human rankings converted to continuous values using a specific scoring script; direct continuous annotation or different normalization will yield different results.
• The evaluation focuses on committee protocols only, excluding plenary sessions to avoid topic and stylistic confounds.
• Hebrew text must be undotted (non-diacriticized) to match the lexicon and model training format.

Evidence (verbatim from paper)

We evaluated inter-annotator agreement by computing the mean pair-wise Pearson correlation between individual annotator scores, after applying the scoring script. Despite the rather subjective nature of this task, we obtained reasonably good inter-annotator agreement scores: 0.904, 0.662 and 0.675 for V, A and D, respectively.

Citation

@misc{goldin2025unveiling,
  title={Unveiling Affective Polarization Trends in Parliamentary Proceedings},
  author={Goldin et al. (2025)},
  year={2025},
  note={arXiv:2512.05231}
}

• arXiv: 2512.05231