Carl Gt Eval

carl-gt-eval

CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models — Tu et al. (2024) (arXiv:2412.17970, 2024)

What this evaluates

Evaluates large language models' causal reasoning capabilities across three task categories: causal graph reasoning (adjacency matrix, d-separation, causal direction), knowledge discovery from tabular data, and decision-making under interventions and counterfactuals.

Datasets

• CARL-GT — total ?; splits: test (-1); repo https://github.com/TURuibo/CauTabBench

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used for adjacency matrix and causal direction estimation.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Used for d-separation estimation (binary classification of d-separated vs d-connected).
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted causal directions in knowledge discovery tasks.
• MAE — range: [0, ∞)
  • Mean Absolute Error between predicted float values and ground-truth expectations for intervention and counterfactual inference.

Input / output format

Input: Text prompts containing serialized causal graphs (node/edge lists) and/or tabular data (Markdown format, 20/50/100 rows), followed by task-specific questions (e.g., d-separation queries, causal direction questions, or intervention/counterfactual value requests).

Output: Free-form natural language responses from the LLM, which are subsequently processed by a second LLM prompt to extract structured answers (e.g., neighbor lists, yes/no, or float numbers).

Scoring recipe

def score(predictions, gold, task):
    if task in ['adjacency', 'direction']:
        pred_set = extract_neighbors(predictions)
        gold_set = gold['edges']
        tp = len(pred_set & gold_set)
        fp = len(pred_set - gold_set)
        fn = len(gold_set - pred_set)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        return {'precision': prec, 'recall': rec, 'f1': f1}
    elif task == 'd-separation':
        labels = [1 if d_sep else 0 for d_sep in gold['is_d_sep']]
        scores = [float(p) for p in predictions]
        return compute_auc(labels, scores)
    elif task in ['intervention', 'counterfactual']:
        preds = [float(p) for p in predictions]
        golds = [float(g) for g in gold['true_expectation']]
        return mean_absolute_error(preds, golds)

Common pitfalls

• LLMs frequently fail to answer knowledge discovery questions, requiring fallback to random yes/no or skipping the instance, which can bias metrics.
• Hardware constraints force different input row limits per model (e.g., Gemma2 restricted to 20 rows, Mixtral excluded), complicating direct performance comparison.
• The two-step evaluation (generation then extraction) can propagate errors if the extraction prompt misinterprets the model's free-form output.

Evidence (verbatim from paper)

As for adjacency matrix and causal direction estimation, LLMs are required to recover 10 causal graphs. The metrics are F1 scores, recall, and precision. As for d-separation estimation, LLMs classify d-separated and d-connected relationships chosen from 10 causal graphs. And the metric is the AUC of ROC curves.

Citation

@misc{tu2024carlgt,
  title={CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models},
  author={Tu et al. (2024)},
  year={2024},
  note={arXiv:2412.17970}
}

• arXiv: 2412.17970