Causalverse Eval

causalverse-eval

CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations — Chen et al. (2025) (arXiv:2510.14049, 2025)

What this evaluates

Probes the ability of causal representation learning (CRL) models to recover ground-truth latent variables from high-fidelity visual simulations. It evaluates both component-wise and block-wise identifiability under realistic conditions where theoretical assumptions may be violated.

Datasets

• CausalVerse — total 200000; splits: test (-1)

Metrics

• Mean Correlation Coefficient (MCC) (primary) — range: [0, 1]
  • Computes Pearson correlations between ground-truth latents Z and estimated latents Z_hat. Selects an injective matching pi maximizing the sum of absolute correlations, then averages them: MCC = (1/D) * sum(|corr(Z_i, Z_hat_{pi(i)})|).
• Coefficient of Determination (R^2) — range: other
  • Measures block-wise identifiability by regressing ground-truth block z_b from estimated block z_hat_b. Formula: R^2 = 1 - Var(z_b - f(z_hat_b)) / Var(z_b), where f is the best linear or non-linear predictor.
• Over-completed MCC — range: [0, 1]
  • Variant of MCC for over-complete settings (D_hat > D). Selects the top D estimated variables that best match ground truth, then applies standard MCC computation over this subset.

Input / output format

Input: Static images or video frames from configurable high-fidelity simulations. Models receive raw visual data without ground-truth latent labels during unsupervised training.

Output: Estimated latent vectors Z_hat in R^{D_hat} (or block-wise estimates z_hat_b) produced by the model's encoder.

Scoring recipe

def compute_mcc(Z, Z_hat):
    corr_matrix = pearson_corr(Z, Z_hat) # D x D_hat
    best_perm = argmax_permutation(corr_matrix, metric=abs)
    return mean(abs(corr_matrix[range(D), best_perm]))

def compute_r2(Z_b, Z_hat_b):
    f = fit_regression(Z_b, Z_hat_b) # linear or non-linear
    return 1 - var(Z_b - f(Z_hat_b)) / var(Z_b)

Common pitfalls

• Assuming CRL methods satisfy theoretical assumptions (e.g., sufficient change, sparsity) when applied to realistic simulated data, leading to misleadingly low MCC scores.
• Confusing component-wise identifiability (MCC) with block-wise identifiability (R^2), as methods may excel at one metric while failing at the other.
• Interpreting negative R^2 values as implementation errors rather than valid indicators that the model performs worse than predicting the mean.

Evidence (verbatim from paper)

To evaluate both component-wise and block-wise identifiability in CausalVerse, we adopt three metrics, including the Mean Correlation Coefficient (MCC), the coefficient of determination $R^{2}$, and the over-completed MCC. Specifically, let $Z\in\mathbb{R}^{D}$ be the ground-truth latent vector and $\widehat{Z}\in\mathbb{R}^{\widehat{D}}$ the estimated vector, we have: Mean Correlation Coefficient (MCC): To calculate MCC, we first compute the Pearson correlations $R_{ij}=\mathrm{corr}(Z_{i},\widehat{Z}{j}),$ then select an injective matching $\pi:{1,\dots,D}\to{1,\dots,\widehat{D}}$ maximizing $\sum{i=1}^{D}|R_{i,\pi(i)}|$. Finally, the MCC value is defined as $\mathrm{MCC}=\frac{1}{D}\sum_{i=1}^{D}\bigl|R_{i,\pi(i)}\bigr|.$

Citation

@misc{chen2025causalverse,
  title={CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2510.14049}
}

• arXiv: 2510.14049