Climatecause Eval

climatecause-eval

ClimateCause: Complex and Implicit Causal Structures in Climate Reports — Allein et al. (2026) (arXiv:2604.14856, 2026)

What this evaluates

Evaluates large language models' ability to infer correlation directions between event pairs and identify causal chain structures (membership and node position) from climate science text, including implicit and nested causal relations.

Datasets

• ClimateCause — total ?; splits: (unstated); repo https://github.com/laallein/ClimateCause

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: $F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}$. Reported as mean and standard deviation across 3 runs per prompt strategy.

Input / output format

Input: Varies by task: (1) CorrI/CorrI+RC: a set of causal pairs $R$, optionally with a statement $s$. (2) CCR: a causal graph with nodes $V$ and edges. (3) CCR+ECI+RC: a statement $s$ and a set of events $V$. Inputs are formatted via zero-shot, few-shot, chain-of-thought, or graph-encoding prompts.

Output: Binary or multi-class labels: correlation direction (positive/negative), chain membership (yes/no), or chain position (start/middle/end/none).

Scoring recipe

def compute_f1(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t != p and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t != p and t == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• LLMs frequently over-predict chain membership and positions, confusing the 'none' class with actual chain nodes.
• Providing the original statement (RC) does not significantly improve correlation inference performance despite high recall.
• Graph-oriented prompt definitions may misalign with text-only inputs, causing abstract causal chain reasoning failures.

Evidence (verbatim from paper)

Table 4: Mean and standard deviation results for each prompting strategy individually (3 runs) and all strategies together (9 runs; Avg). | Prompt | Precision | Recall | F1 |

Citation

@misc{allein2026climatecause,
  title={ClimateCause: Complex and Implicit Causal Structures in Climate Reports},
  author={Allein et al. (2026)},
  year={2026},
  note={arXiv:2604.14856}
}

• arXiv: 2604.14856