Eci Eval

eci-eval

A Survey of Event Causality Identification: Taxonomy, Challenges, Assessment, and Prospects — Qing Cheng et al. (2024) (arXiv:2411.10371, 2024)

What this evaluates

This evaluation protocol assesses the capability of NLP models to identify causal relationships between event pairs in text. It probes both sentence-level and document-level reasoning, measuring how well models can distinguish true causal links from mere correlations or coincidental co-occurrences.

Datasets

• CTB — total ?; splits: test (-1)
• ESL — total ?; splits: dev (-1), train (-1)
• MAVEN-ERE — total ?; splits: test (-1), val (-1)
• MECI — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: percent
  • F1 = 2 * (Precision * Recall) / (Precision + Recall). Precision is the proportion of predicted causal pairs that are correct, and Recall is the proportion of actual causal pairs that are correctly identified.
• Precision — range: percent
  • Precision = TP / (TP + FP). Measures the proportion of predicted causal relationships that are actually correct.
• Recall — range: percent
  • Recall = TP / (TP + FN). Measures the proportion of actual causal relationships that are correctly identified by the model.

Input / output format

Input: For sentence-level tasks: [Sentence] [SEP] Is there a causal relationship between [Event1] and [Event2]? For document-level tasks: [Document] [SEP] Is there a causal relationship between [Event1] and [Event2]?

Output: Binary classification output (Yes/No or direct generation) indicating the presence or absence of a causal relationship between the specified event pair.

Scoring recipe

tp = sum(1 for pred, gold in zip(predictions, golds) if pred == 1 and gold == 1)
fp = sum(1 for pred, gold in zip(predictions, golds) if pred == 1 and gold == 0)
fn = sum(1 for pred, gold in zip(predictions, golds) if pred == 0 and gold == 1)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return {'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Directional accuracy is explicitly ignored; only the existence of a causal link is evaluated.
• LLMs often suffer from 'causal hallucination,' producing high recall but low precision due to over-predicting causal links.
• Dataset splits and cross-validation schemes vary significantly across benchmarks (e.g., 10-fold vs 5-fold vs fixed dev/test), complicating direct performance comparisons.
• Some baseline models use distinct data processing methods or different splits than reported, requiring careful reproduction or direct citation to maintain fairness.

Evidence (verbatim from paper)

We evaluated the models using Precision (P), Recall (R), and F1-score (F1) as metrics, distinguishing between intra-sentence causality, inter-sentence causality, and overall performance. Given that most existing methods evaluate only the presence of causalities without considering directional accuracy, we focused our comparison similarly on causality existence alone.

Citation

@misc{cheng2024survey,
  title={A Survey of Event Causality Identification: Taxonomy, Challenges, Assessment, and Prospects},
  author={Qing Cheng et al. (2024)},
  year={2024},
  note={arXiv:2411.10371}
}

• arXiv: 2411.10371