Causal3d Eval

causal3d-eval

CAUSAL3D: A Comprehensive Benchmark for Causal Learning from Visual Data — Liu et al. (2025) (arXiv:2503.04852, 2025)

What this evaluates

This benchmark evaluates a model's ability to discover and reason about causal structures from visual and tabular data. It probes whether models can infer correct causal graphs from observational data, learn disentangled representations from images, and perform valid causal interventions with limited visual samples.

Datasets

• Causal3D — total 190000; splits: test (-1)

Metrics

• correctness of inferred causal structures (primary) — range: [0, 1]
  • Measures the accuracy of the predicted causal graph against the ground truth DAG. Evaluated by comparing inferred edges and causal mechanisms across datasets.
• intervention consistency — range: [0, 1]
  • Assesses whether intervened images remain consistent with the underlying causal relations after applying the do-operator to a specific variable.

Input / output format

Input: Tabular data containing variable values for causal discovery; or multi-perspective 3D rendered images (with/without real/virtual backgrounds) for representation learning and few-shot intervention tasks.

Output: Inferred causal graph/structure (for discovery tasks); or intervened/generated images reflecting the manipulated causal variables (for representation and intervention tasks).

Scoring recipe

def score_causal_discovery(predictions, ground_truth):
    pred_edges = set(predictions.edges)
    gt_edges = set(ground_truth.edges)
    correct = len(pred_edges.intersection(gt_edges))
    return correct / len(gt_edges)

def score_intervention(intervened_images, ground_truth_relations):
    consistent_count = 0
    for img in intervened_images:
        if check_causal_consistency(img, ground_truth_relations):
            consistent_count += 1
    return consistent_count / len(intervened_images)

Common pitfalls

• Models may overfit to specific backgrounds or camera views rather than learning invariant causal relations.
• Causal discovery from few images is highly sensitive to initialization and may fail to converge without strong inductive biases.
• The benchmark mixes linear and nonlinear causal mechanisms, which can cause standard linear causal discovery algorithms to fail unexpectedly.

Evidence (verbatim from paper)

These tasks include: Causal discovery from tabular data... evaluated based on the correctness of inferred causal structures across various datasets and underlying causal mechanisms. ... Causal representation learning from images... Evaluation is based on generated images after intervening on learned representations, assessing whether they accurately reflect the corresponding causal variables and causal relations. ... Causal discovery & intervention from few images... Intervention evaluation is based on whether the intervened images still remain consistent with the underlying causal relations.

Citation

@misc{liu2025causal3d,
  title={CAUSAL3D: A Comprehensive Benchmark for Causal Learning from Visual Data},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2503.04852}
}

• arXiv: 2503.04852