Vihos Eval

vihos-eval

ViHOS: Hate Speech Spans Detection for Vietnamese — Hoang et al. (2023) (arXiv:2301.10186, 2023)

What this evaluates

Evaluates a model's ability to detect hate and offensive text spans within Vietnamese social media comments. It probes sequence tagging capabilities, specifically requiring precise boundary identification of offensive content in noisy, informal text.

Datasets

• ViHOS — total 11056; splits: test (-1), dev (-1); repo https://github.com/phusroyal/ViHOS

Metrics

• macro-average F1-score (primary) — range: [0, 1]
  • Computed as the arithmetic mean of F1 scores across all gold-predicted span pairs. Precision, Recall, and F1 are calculated at the span level, then averaged across the dataset.

Input / output format

Input: Vietnamese social media comments (text), tokenized at either syllable-level or word-level.

Output: IOB (Inside-Outside-Beginning) character-level tags indicating the start and end boundaries of hate/offensive spans.

Scoring recipe

def compute_span_f1(gold_spans, pred_spans):
    tp = len(set(gold_spans) & set(pred_spans))
    fp = len(set(pred_spans) - set(gold_spans))
    fn = len(set(gold_spans) - set(pred_spans))
    p = tp / (tp + fp) if (tp + fp) > 0 else 0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0

macro_f1 = sum(compute_span_f1(g, p) for g, p in zip(gold_spans_list, pred_spans_list)) / len(gold_spans_list)

Common pitfalls

• Tokenization granularity (syllable vs. word) drastically changes performance depending on the model architecture.
• Censored text (e.g., asterisks like 'b**i') and informal social media formatting often cause incorrect span boundary predictions.
• Context-dependent metaphors, idioms, and allusions lead to frequent false negatives even for strong pre-trained models.

Evidence (verbatim from paper)

The macro-average F1-score (F1) is used to evaluate our models. For each pair of gold-predicted spans, we compute F1 and then calculate the arithmetic mean of F1 for each of these cases. It should be noted that the final F1-score, Accuracy, and Precision reported are an average of more than ten runs with various random seeds.

Citation

@misc{hoang2023vihos,
  title={ViHOS: Hate Speech Spans Detection for Vietnamese},
  author={Hoang et al. (2023)},
  year={2023},
  note={arXiv:2301.10186}
}

• arXiv: 2301.10186