Polar Detection Eval

polar-detection-eval

BITS Pilani at SemEval-2026 Task 9: Structured Supervised Fine-Tuning with DPO Refinement for Polarization Detection — Gupta et al. (2026) (arXiv:2604.11121, 2026)

What this evaluates

This benchmark evaluates a model's ability to detect online polarization in social media text by classifying statements as polarized or non-polarized. It specifically probes the model's capacity to produce interpretable, structured reasoning alongside binary predictions while handling class imbalance and reducing false negatives.

Datasets

• POLAR @ SemEval-2026 — total 4834; splits: train (3222), dev (160), test (1452)

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across the polarized and non-polarized classes. Calculated as the unweighted mean of the F1 scores for each class, where F1 = 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Raw input text, optionally with referenced target and claim type. For inference, formatted as 'Input: {text}\nReasoning:'.

Output: A structured completion containing fields for target, claim type, manifestations (present/absent checklist), decision basis, and a final binary label (0 or 1). The label is extracted via regex from the generated text.

Scoring recipe

def compute_macro_f1(predictions, golds):
    tp = sum(1 for p, g in zip(predictions, golds) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, golds) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, golds) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(predictions, golds) if p == 0 and g == 0)
    prec_pos = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec_pos = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1_pos = 2 * prec_pos * rec_pos / (prec_pos + rec_pos) if (prec_pos + rec_pos) > 0 else 0.0
    prec_neg = tn / (tn + fn) if (tn + fn) > 0 else 0.0
    rec_neg = tn / (tn + fp) if (tn + fp) > 0 else 0.0
    f1_neg = 2 * prec_neg * rec_neg / (prec_neg + rec_neg) if (prec_neg + rec_neg) > 0 else 0.0
    return (f1_pos + f1_neg) / 2.0

Common pitfalls

• Regex-based label extraction can fail if the model deviates from the strict structured template, leading to dropped predictions.
• The dataset is imbalanced (~36.5% polarized), making false negatives a critical failure mode that standard SFT struggles with.
• Evaluation focuses on the English subset; multilingual performance is not reported in this specific run.

Evidence (verbatim from paper)

We establish a simple fine-tuning baseline by splitting the data into train/validation/test (80/10/10), fine-tuning a classifier with LoRA adapters, and evaluating with macro precision, recall, and F1.

Citation

@misc{gupta2026bitspilani,
  title={BITS Pilani at SemEval-2026 Task 9: Structured Supervised Fine-Tuning with DPO Refinement for Polarization Detection},
  author={Gupta et al. (2026)},
  year={2026},
  note={arXiv:2604.11121}
}

• arXiv: 2604.11121