Multiparadetox Eval

multiparadetox-eval

MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages — Dementieva et al. (2024) (arXiv:2404.02037, 2024)

What this evaluates

Evaluates text detoxification models across Russian, Ukrainian, and Spanish by measuring how effectively they transform toxic input into neutral output. The benchmark probes a model's ability to remove offensive language while preserving the original semantic content and maintaining grammatical fluency in the target language.

Datasets

• MultiParaDetox — total ?; splits: test (-1)

Metrics

• STA (primary) — range: [0, 1]
  • Style Transfer Accuracy; measures the proportion of toxic text successfully transformed into neutral register. Exact computation formula not provided in the text.
• SIM — range: [0, 1]
  • Content Similarity; measures how well the detoxified output preserves the original semantic content. Exact computation formula not provided in the text.
• FL — range: [0, 1]
  • Fluency; measures the grammatical correctness and naturalness of the generated detoxified text. Exact computation formula not provided in the text.
• J — range: [0, 1]
  • J-score; a composite metric used to compare unsupervised baselines against fine-tuned models. Exact computation formula not provided in the text.

Input / output format

Input: Toxic text in a target language (Russian, Ukrainian, Spanish, or multilingual).

Output: Detoxified (neutral register) text in the same language.

Scoring recipe

def evaluate(predictions, references):
    # Metrics computed per instance, then averaged per language/method
    sta_scores = [compute_sta(pred, ref) for pred, ref in zip(predictions, references)]
    sim_scores = [compute_sim(pred, ref) for pred, ref in zip(predictions, references)]
    fl_scores = [compute_fl(pred) for pred in predictions]
    j_scores = [compute_j(sta, sim, fl) for sta, sim, fl in zip(sta_scores, sim_scores, fl_scores)]
    return {
        'STA': mean(sta_scores),
        'SIM': mean(sim_scores),
        'FL': mean(fl_scores),
        'J': mean(j_scores)
    }

Common pitfalls

• High STA scores may result from model hallucination or generating text in the wrong language rather than genuine detoxification.
• Direct translation of English toxic phrases fails in morphologically rich languages like Russian and Ukrainian due to cultural and linguistic specificity.
• Performance drops (e.g., in Spanish) can be driven by significantly smaller training data sizes rather than model deficiency.

Evidence (verbatim from paper)

Delete methods reaches the highest content similarity as it was designed to modify the original sentence slightly. However, it does not filter all toxic language and gains the lowest STA scores. The condBERT method fails to make substitutions with correct words and obtains not good enough fluency scores. LLaMa achieves very high STA scores concurrently with the lowest SIM scores. The model can hallucinate and even generate text not in a target language as can be observed from the examples.

Citation

@misc{dementieva2024multiparadetox,
  title={MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages},
  author={Dementieva et al. (2024)},
  year={2024},
  note={arXiv:2404.02037}
}

• arXiv: 2404.02037