Multilingual Safety Eval

multilingual-safety-eval

CREST: Universal Safety Guardrails Through Cluster-Guided Cross-Lingual Transfer — Bansal et al. (2025) (arXiv:2512.02711, 2025)

What this evaluates

Evaluates a parameter-efficient multilingual safety guardrail's ability to classify content as safe or unsafe across high-resource and low-resource languages. It probes cross-lingual generalization and robustness against diverse harm categories using cluster-guided transfer.

Datasets

• Aegis-Content-Safety-2.0-Test (Aegis-CS2) — total ?; splits: test (-1)
• HarmBench — total ?; splits: test (-1)
• Redteam2k — total ?; splits: test (-1)
• JBB-Behaviors — total ?; splits: test (-1)
• StrongReject — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard binary classification accuracy: the proportion of correctly classified safe/unsafe instances out of the total test instances.

Input / output format

Input: Natural language prompts or conversation turns in 24 languages (13 In-Domain, 11 Out-of-Domain), representing human-LLM interactions or red-teaming queries.

Output: Binary classification label (safe vs. unsafe) produced by a single-layer classification head.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Evaluating on Out-of-Domain (OOD) low-resource languages requires careful handling of translation artifacts, as training data was translated from English to In-Domain languages.
• Benchmarks cover diverse harm categories (hate, sexual, violence, etc.), so aggregate scores may mask per-category performance gaps.
• Cross-lingual transfer relies on linguistic clustering; performance may drop sharply for languages outside the 100-language XLM-R pretraining coverage.

Evidence (verbatim from paper)

For evaluation, we benchmark our model on six safety classification datasets: Aegis-Content-Safety-2.0-Test (Aegis-CS2) (2025), HarmBench (Mazeika et al. 2024), Redteam2k (Luo et al. 2024), JBB-Behaviors (subsets Behaviors as JBB-Behav and Judge-comparison as JBB-Judge) (Chao et al. 2024), and StrongReject (Souly et al. 2024). These benchmarks collectively span various harm categories, including but not limited to Hate/Identity Hate, Sexual, Suicide/Self-Harm, Violence, Guns/Illegal Weapons, PII/Privacy, Sexual Minor, Toxicity, Abuse, etc., which makes them suitable for a comprehensive evaluation of safety guardrails.

Citation

@misc{bansal2025crest,
  title={CREST: Universal Safety Guardrails Through Cluster-Guided Cross-Lingual Transfer},
  author={Bansal et al. (2025)},
  year={2025},
  note={arXiv:2512.02711}
}

• arXiv: 2512.02711