Counterfactual Text Gen Eval

counterfactual-text-gen-eval

CEval: A Benchmark for Evaluating Counterfactual Text Generation — Van Bach Nguyen et al. (2024) (arXiv:2404.17475, 2024)

What this evaluates

This benchmark evaluates the effectiveness and linguistic quality of counterfactual text generation methods. It probes a model's ability to modify input text to flip a target classifier's predicted label while preserving grammatical correctness, fluency, and coherence, highlighting the trade-off between label-flipping success and text quality.

Datasets

• IMDB — total ?; splits: test (-1); repo https://github.com/aix-group/CEval-Counterfactual-Generation-Benchmark
• SNLI — total ?; splits: test (-1); repo https://github.com/aix-group/CEval-Counterfactual-Generation-Benchmark

Metrics

• flip rate (FR) (primary) — range: [0, 1]
  • The proportion of generated counterfactuals that successfully change the target classifier's predicted label compared to the original text. Calculated as (number of successful flips) / (total generated instances).
• text quality (Grammar, Cohesiveness, Fluency) — range: [1, 5]
  • Scores assigned by LLM judges (GPT-3.5 Turbo and Mistral) evaluating grammatical correctness, logical flow, and naturalness of the generated text. Typically rated on a Likert-like scale.
• perplexity — range: positive real
  • A measure of how well a probability model predicts a sample, computed here using GPT-2. Lower values indicate more fluent and predictable text.
• edit distance (Levenshtein) — range: non-negative integer
  • The minimum number of single-character edits (insertions, deletions, or substitutions) required to change the original text into the generated counterfactual.
• diversity — range: non-negative
  • A measure of variation across generated counterfactuals, often calculated via unique token/n-gram counts or type-token ratios.
• probability change ($\Delta$P) — range: [0, 1]
  • The absolute difference in the target classifier's predicted probability for the original label versus the counterfactual label. Indicates how far the generation pushes the instance away from the decision boundary.

Input / output format

Input: Original text instance and a prompt instructing the model to generate a counterfactual version that flips the target classifier's label.

Output: A single generated counterfactual text string.

Scoring recipe

def score_counterfactuals(predictions, golds, target_classifier):
    flip_count = 0
    ed_scores = []
    for pred, gold in zip(predictions, golds):
        if target_classifier.predict(pred) != target_classifier.predict(gold):
            flip_count += 1
        ed_scores.append(levenshtein_distance(gold, pred))
    fr = flip_count / len(predictions)
    avg_ed = sum(ed_scores) / len(ed_scores)
    # LLM quality scores computed via separate API calls per instance
    return {'flip_rate': fr, 'avg_edit_distance': avg_ed}

Common pitfalls

• Optimizing solely for label flipping often degrades text quality, grammar, and fluency; the benchmark explicitly highlights this trade-off rather than treating flip rate as the sole success metric.
• LLM-based evaluators (GPT, Mistral) may exhibit bias toward LLM-generated outputs over human-written text, skewing quality scores upward for models like LLAMA-2.
• High diversity scores strongly correlate with high edit distance (r=0.93), so they should not be interpreted as independent measures of generation variation.

Evidence (verbatim from paper)

Interestingly, MICE has the highest flip rate (FR), but not the largest change in target label probability change ($\Delta$P) on the IMDB dataset.

Citation

@misc{nguyen2024ceval,
  title={CEval: A Benchmark for Evaluating Counterfactual Text Generation},
  author={Van Bach Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2404.17475}
}

• arXiv: 2404.17475