E Care Eval

e-care-eval

e-CARE: a New Dataset for Exploring Explainable Causal Reasoning — Li Du et al. (2022) (arXiv:2205.05849, 2022)

What this evaluates

Evaluates a model's ability to perform commonsense causal reasoning by predicting the reasonableness of causal facts, and to generate conceptually grounded natural language explanations for those causal relationships.

Datasets

• e-CARE — total 21000; splits: test (-1); repo https://github.com/Waste-Wood/e-CARE

Metrics

• Accuracy (%) (primary) — range: percent
  • Standard classification accuracy calculated as the proportion of correctly predicted causal fact reasonableness labels out of the total test instances.
• CEQ score — range: [0, 1]
  • A novel metric proposed by the authors to evaluate explanation quality by assessing how well explanations promote understanding of causal mechanisms, rather than relying on textual similarity.
• Human Evaluation (%) — range: percent
  • Percentage of model-generated explanations labeled as valid by three human annotators who verify if the explanation correctly explains the corresponding causal fact.
• AVG-BLEU — range: [0, 1]
  • Average n-gram precision (n=4) between generated explanations and human-written reference explanations, averaged across all test instances.
• ROUGE-1 — range: [0, 1]
  • Recall of unigram overlap between generated explanations and reference explanations.
• ΔAccu. (%) after Adv. Attack — range: percent
  • Change in causal reasoning accuracy after applying a gradient-based FGM adversarial perturbation to the input word embeddings.

Input / output format

Input: Causal reasoning: A candidate causal fact composed of a premise and a corresponding hypothesis. Explanation generation: Concatenation of the cause and effect.

Output: Causal reasoning: A score or label measuring the reasonableness of the candidate causal fact. Explanation generation: A natural language conceptual explanation.

Scoring recipe

def evaluate_causal_reasoning(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels) * 100

def evaluate_explanations(generated_exps, gold_exps, human_labels):
    bleu = compute_avg_bleu4(generated_exps, gold_exps)
    rouge1 = compute_rouge1(generated_exps, gold_exps)
    valid_count = sum(1 for exp in generated_exps if human_labels[exp] == 'valid')
    human_acc = valid_count / len(generated_exps) * 100
    return bleu, rouge1, human_acc

Common pitfalls

• Models may exploit superficial lexical cues rather than genuine causal understanding, leading to inflated performance on smaller benchmarks.
• Standard text-similarity metrics (AVG-BLEU, ROUGE-1) show near-zero correlation with human judgments of explanation quality, making them unreliable for this task.

Evidence (verbatim from paper)

We cast the causal reasoning task as a prediction problem: The input of the model is a candidate causal fact composed of a premise and one of the corresponding candidate hypotheses. The output is a score measuring the reasonableness of the candidate causal fact. We automatically evaluate the quality of generated explanations using average-BLEU (n = 4) (Papineni et al., 2002), ROUGE-1 (Lin, 2004), Perplexity (Horgan, 1995), together with our proposed CEQ score.

Citation

@misc{du2022ecare,
  title={e-CARE: a New Dataset for Exploring Explainable Causal Reasoning},
  author={Li Du et al. (2022)},
  year={2022},
  note={arXiv:2205.05849}
}

• arXiv: 2205.05849