Countereval Eval

countereval-eval

Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments — Domnich et al. (2024) (arXiv:2410.21131, 2024)

What this evaluates

Evaluates the human-centric quality of counterfactual explanations across eight explanatory virtues. It probes how well explanations convey desired outcomes, remain feasible, consistent, complete, trustworthy, understandable, fair, and appropriately complex.

Datasets

• CounterEval — total 30; splits: full (30); HF anitera/CounterEval; repo https://github.com/anitera/CounterEval

Metrics

• Overall Satisfaction, Feasibility, Consistency, Completeness, Trust, Understandability, Fairness, Complexity (primary) — range: 1-6 (or -2 to 2 for Complexity)
  • Ordinal rating assigned by human evaluators on a 6-point scale (1-6), except Complexity on a 5-point scale (-2 to 2, where 0 is ideal). LLM performance is measured by exact-match accuracy in predicting these human-assigned scores.

Input / output format

Input: Textual counterfactual explanation scenarios describing feature changes to achieve a different outcome, accompanied by contextual information and specific evaluation questions.

Output: Ordinal score from 1 to 6 for each of the 8 metrics (or -2 to 2 for Complexity), representing the evaluator's agreement or assessment of the explanation's quality on that dimension.

Scoring recipe

def compute_prediction_accuracy(predictions, gold):
    # predictions and gold are lists of human-assigned scores per instance
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(predictions)

Common pitfalls

• Complexity uses a different scale (-2 to 2) where 0 is ideal, unlike the other 7 metrics which use a 1-6 scale.
• Human ratings are collected via survey and averaged per question; LLMs must predict these ordinal scores rather than continuous values.
• The dataset is small (30 instances) and designed to cover extreme variations across metrics, so results may not generalize to typical real-world explanations.

Evidence (verbatim from paper)

All metrics were rated on scales detailed in Table[1], with a 6-point ordinal scale from 1 (lowest) to 6 (highest) except for Complexity, rated on a 5-point scale from -2 (too simple) to 2 (too complex), where 0 corresponded to desired complexity.

Citation

@misc{domnich2024countereval,
  title={Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments},
  author={Domnich et al. (2024)},
  year={2024},
  note={arXiv:2410.21131}
}

• arXiv: 2410.21131