Swsr Eval

swsr-eval

SWSR: A Chinese Dataset and Lexicon for Online Sexism Detection — Jiang et al. (2021) (arXiv:2108.03070, 2021)

What this evaluates

Probes the capability of NLP models to detect online sexism in Chinese microblogging comments. It evaluates performance across three hierarchical classification tasks: binary sexism identification, fine-grained category classification, and target type classification.

Datasets

• SWSR — total 8969; splits: train (-1), test (-1)

Metrics

• macro F1 (primary) — range: [0, 1]
  • The unweighted mean of the F1 scores computed for each class independently. F1 for a class is 2 * (precision * recall) / (precision + recall), where precision and recall are calculated per class.
• accuracy — range: [0, 1]
  • The proportion of correctly classified instances out of the total number of instances.
• weighted F1 — range: [0, 1]
  • The mean of F1 scores weighted by the number of true instances for each class, accounting for class imbalance.

Input / output format

Input: Raw Chinese text of online comments or weibo posts.

Output: Discrete class label corresponding to the task: 'sexist' or 'non-sexist' (binary); 'SA', 'SCB', 'MA', 'SO', or 'non-sexist' (category); or 'generic', 'individual', or 'non-sexist' (target).

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    classes = sorted(set(golds) | set(preds))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    return macro_f1, acc

Common pitfalls

• The dataset is highly imbalanced across fine-grained classes, making accuracy and weighted F1 potentially misleading; macro F1 is explicitly reported to mitigate this bias.
• Chinese text lacks explicit word boundaries, so character-level features or models are required for competitive performance, unlike standard word-level n-gram baselines.
• Lexicon integration uses TF-IDF on raw word counts concatenated with embeddings, which may not align well with contextual transformer representations and yields only marginal gains.

Evidence (verbatim from paper)

We report global macro F1 and accuracy scores for the three tasks, as well as F1 scores specific to each class for experimental step 1 and weighted F1 scores for steps 2 and 3.

Citation

@misc{jiang2021swsr,
  title={SWSR: A Chinese Dataset and Lexicon for Online Sexism Detection},
  author={Jiang et al. (2021)},
  year={2021},
  note={arXiv:2108.03070}
}

• arXiv: 2108.03070