# Ukrainian Text Classification Eval

> Evaluates cross-lingual transfer methods for Ukrainian text classification across toxicity, formality, and natural language inference tasks. It compares translation-based baselines, LLM prompting, and adapter/fine-tuning approaches on both machine-translated and semi-natural Ukrainian test sets. Use when the user wants to benchmark on Ukrainian Toxicity (Translated & Semi-natural), Ukrainian Formality (Translated & Semi-natural), Ukrainian NLI (Translated & Semi-natural), or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/ukrainian-text-classification-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/ukrainian-text-classification-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/ukrainian-text-classification-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/ukrainian-text-classification-eval

---


# ukrainian-text-classification-eval

> Cross-lingual Text Classification Transfer: The Case of Ukrainian — Dementieva et al. (2024) (arXiv:2404.02043, 2024)

## What this evaluates

Evaluates cross-lingual transfer methods for Ukrainian text classification across toxicity, formality, and natural language inference tasks. It compares translation-based baselines, LLM prompting, and adapter/fine-tuning approaches on both machine-translated and semi-natural Ukrainian test sets.

## Datasets

- **Ukrainian Toxicity (Translated & Semi-natural)** — total ?; splits: test (-1)
- **Ukrainian Formality (Translated & Semi-natural)** — total ?; splits: test (-1)
- **Ukrainian NLI (Translated & Semi-natural)** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels out of all predictions.
- `Precision` — range: [0, 1]
  - Ratio of true positive predictions to all positive predictions.
- `Recall` — range: [0, 1]
  - Ratio of true positive predictions to all actual positive instances.
- `F1` — range: [0, 1]
  - Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).

## Input / output format

**Input**: Raw Ukrainian text for classification tasks. For backtranslation baselines, English text is provided to the translation system first.

**Output**: Predicted class label: 'toxic' or 'non-toxic' for toxicity; 'formal' or 'informal' for formality; 'entailment', 'neutral', or 'contradiction' for NLI.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    correct = sum(p == g for p, g in zip(preds, golds))
    acc = correct / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g == 'pos')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'pos' and g != 'pos')
    fn = sum(1 for p, g in zip(preds, golds) if p != 'pos' and g == 'pos')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return acc, prec, rec, f1
```

## Common pitfalls

- Backtranslation baseline is only evaluated on the semi-natural test set, as the translated test set column shows '—' for this method.
- Toxicity labels from Jigsaw are collapsed into a binary 'toxic'/'non-toxic' scheme, which may obscure nuances present in the original multi-class dataset.
- Semi-natural test sets are constructed from heterogeneous sources (tweets, legal acts, fiction), potentially introducing domain bias not present in the translated benchmarks.

## Evidence (verbatim from paper)

> |  | Acc | Pr | Re | F1 | Acc | Pr | Re | F1 |
Табл. 3: Ukrainian Texts Classification results. We divide methods into two groups – not requiring and requiring fine-tuning. Then, bold numbers denote the best results within the methods group and a test set, underline – overall best scores for the task.

## Citation

```bibtex
@misc{dementieva2024crosslingual,
  title={Cross-lingual Text Classification Transfer: The Case of Ukrainian},
  author={Dementieva et al. (2024)},
  year={2024},
  note={arXiv:2404.02043}
}
```

- arXiv: 2404.02043