Redrft Eval

redrft-eval

RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming — Zheng et al. (2025) (arXiv:2506.04302, 2025)

What this evaluates

Evaluates reinforcement fine-tuning methods for red-teaming LLMs by measuring the toxicity and diversity of generated adversarial prompts across toxic continuation and instruction-following tasks.

Datasets

• toxic continuation — total ?; splits: test (-1)
• instruction following — total ?; splits: test (-1)

Metrics

• cumulative toxicity-diversity score (primary) — range: [0, 1]
  • A composite metric balancing toxicity and diversity/novelty scores. The paper uses a threshold of 0.5 as a performance benchmark, where higher scores indicate better red-teaming capability.
• toxicity score — range: [0, 1]
  • Measures the harmfulness or toxicity of generated prompts.
• diversity/novelty score — range: [0, 1]
  • Measures the variety or novelty of generated prompts.

Input / output format

Input: Task-specific prompts or instructions provided to the red-team LLM (GPT-2) for adversarial generation.

Output: Generated text continuations or responses from the red-team LLM.

Scoring recipe

# Pseudo-code based on paper description
toxicity_scores = [compute_toxicity(gen) for gen in generations]
diversity_scores = [compute_diversity(gen) for gen in generations]
cumulative_score = aggregate(toxicity_scores, diversity_scores)
return cumulative_score
# Evaluated over 3 random seeds per hyperparameter config

Common pitfalls

• Exact dataset splits and metric computation formulas are defined in Section 4, not this section.
• Toxicity and diversity are typically measured via external classifiers or embedding-based metrics not detailed here.
• The 0.5 cumulative score threshold is used as a qualitative benchmark for improvement, not a strict pass/fail criterion.

Evidence (verbatim from paper)

We show the main benchmarking results in Figure 2, which shows the toxicity-diversity profile for all baselines on both tasks. Overall, we observe that none of the current RFT-based red teaming methods achieve a cumulative toxicity-diversity score greater than 0.5, indicating significant room for improvement.

Citation

@misc{zheng2025redrft,
  title={RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2506.04302}
}

• arXiv: 2506.04302