Headlinecause Eval

headlinecause-eval

HeadlineCause: A Dataset of News Headlines for Detecting Causalities — Ilya Gusev, Alexey Tikhonov (2021) (arXiv:2108.12626, 2021)

What this evaluates

Evaluates a model's ability to detect implicit causal relationships between two news headlines without relying on explicit causal linking words. It probes commonsense reasoning and world knowledge to distinguish between causal, refutational, same-event, and unrelated headline pairs.

Datasets

• HeadlineCause — total ?; splits: test (967); repo https://github.com/IlyaGusev/HeadlineCause

Metrics

• causality ROC AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve for the binary causal vs non-causal classification task. Calculated by combining Left-right and Right-left causality classes into a single positive class to allow threshold variation.
• F-score — range: [0, 1]
  • Per-class F1-score for the multiclass Full task, computed across seven relationship categories including causality, refutation, and non-causal relations.
• Total multiclass accuracy — range: [0, 1]
  • Percentage of correctly classified headline pairs across all seven relationship categories in the Full task.

Input / output format

Input: A pair of news headline strings (one cause, one effect, or vice versa) in either English or Russian.

Output: A categorical label indicating the relationship type: binary (Causal vs Non-causal) for the Simple task, or multiclass (No relationship, Same event, Other relationship, Left-right causality, Right-left causality, Left-right refutation, Right-left refutation) for the Full task.

Scoring recipe

def score(predictions, gold, task='simple'):
    if task == 'simple':
        y_true = [1 if g in ['Left-right causality', 'Right-left causality'] else 0 for g in gold]
        y_pred = [1 if p in ['Left-right causality', 'Right-left causality'] else 0 for p in predictions]
        return roc_auc_score(y_true, y_pred)
    else:
        acc = accuracy_score(gold, predictions)
        f1s = f1_score(gold, predictions, average=None, zero_division=0)
        return acc, f1s

Common pitfalls

• The dataset specifically targets implicit causality; models relying on explicit causal connectors (e.g., 'because', 'therefore') will artificially inflate performance.
• Refutation classes are extremely imbalanced in the test set (e.g., only 5-8 samples per language), making per-class F1 scores for refutations statistically unreliable.
• The Simple task merges Left-right and Right-left causality into a single positive class, so evaluators must not report separate F1 scores for directionality when evaluating the primary metric.

Evidence (verbatim from paper)

For this task, we consider causality ROC AUC on two classes as a main metric. To calculate it, we unite Left-right and Right-left classes to be able to vary a classifier threshold.

Citation

@misc{gusev2021headlinecause,
  title={HeadlineCause: A Dataset of News Headlines for Detecting Causalities},
  author={Ilya Gusev, Alexey Tikhonov (2021)},
  year={2021},
  note={arXiv:2108.12626}
}

• arXiv: 2108.12626