Ukrainian Text Classification Eval

ukrainian-text-classification-eval

Cross-lingual Text Classification Transfer: The Case of Ukrainian — Dementieva et al. (2024) (arXiv:2404.02043, 2024)

What this evaluates

Evaluates cross-lingual transfer methods for Ukrainian text classification across toxicity, formality, and natural language inference tasks. It compares translation-based baselines, LLM prompting, and adapter/fine-tuning approaches on both machine-translated and semi-natural Ukrainian test sets.

Datasets

• Ukrainian Toxicity (Translated & Semi-natural) — total ?; splits: test (-1)
• Ukrainian Formality (Translated & Semi-natural) — total ?; splits: test (-1)
• Ukrainian NLI (Translated & Semi-natural) — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels out of all predictions.
• Precision — range: [0, 1]
  • Ratio of true positive predictions to all positive predictions.
• Recall — range: [0, 1]
  • Ratio of true positive predictions to all actual positive instances.
• F1 — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: Raw Ukrainian text for classification tasks. For backtranslation baselines, English text is provided to the translation system first.

Output: Predicted class label: 'toxic' or 'non-toxic' for toxicity; 'formal' or 'informal' for formality; 'entailment', 'neutral', or 'contradiction' for NLI.

Scoring recipe

def compute_metrics(preds, golds):
    correct = sum(p == g for p, g in zip(preds, golds))
    acc = correct / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g == 'pos')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'pos' and g != 'pos')
    fn = sum(1 for p, g in zip(preds, golds) if p != 'pos' and g == 'pos')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return acc, prec, rec, f1

Common pitfalls

• Backtranslation baseline is only evaluated on the semi-natural test set, as the translated test set column shows '—' for this method.
• Toxicity labels from Jigsaw are collapsed into a binary 'toxic'/'non-toxic' scheme, which may obscure nuances present in the original multi-class dataset.
• Semi-natural test sets are constructed from heterogeneous sources (tweets, legal acts, fiction), potentially introducing domain bias not present in the translated benchmarks.

Evidence (verbatim from paper)

| | Acc | Pr | Re | F1 | Acc | Pr | Re | F1 | Табл. 3: Ukrainian Texts Classification results. We divide methods into two groups – not requiring and requiring fine-tuning. Then, bold numbers denote the best results within the methods group and a test set, underline – overall best scores for the task.

Citation

@misc{dementieva2024crosslingual,
  title={Cross-lingual Text Classification Transfer: The Case of Ukrainian},
  author={Dementieva et al. (2024)},
  year={2024},
  note={arXiv:2404.02043}
}

• arXiv: 2404.02043