qhjqhj00 counterfactual-chaos-eval
When Counterfactual Reasoning Fails: Chaos and Real-World Complexity — Aalaila et al. (2025) (arXiv:2503.23820, 2025)
What this evaluates
Evaluates the reliability of counterfactual trajectory estimation in chaotic versus non-chaotic dynamical systems under parameter uncertainty and observational noise. It probes whether Bayesian filtering and particle-based smoothing can accurately recover 'what-if' scenarios when small initial perturbations lead to divergent outcomes.
Datasets
- Lorenz System — total ?; splits: (unstated)
- Rössler System — total ?; splits: (unstated)
- Logistic Growth — total ?; splits: (unstated)
Metrics
RMSE_t(primary) — range: [0, ∞)- Root mean squared error over time steps t measuring the average Euclidean distance in d-dimensional phase space between the deterministic counterfactual state X^cf_t and N_cf sampled counterfactual trajectories X^cf_i_t.
Input / output format
Input: Noisy observations of system states over time, process noise variance, observational noise variance, and prior distributions for system parameters. Initial conditions X_0 and counterfactual interventions X^cf_0 are specified.
Output: A set of N_cf counterfactual state trajectories X^cf_i_t for t=1...T in d-dimensional phase space.
Scoring recipe
def compute_rmse_t(X_cf_true, X_cf_samples, d, N_cf):
# X_cf_true: shape (T, d)
# X_cf_samples: shape (N_cf, T, d)
distances = np.sqrt(np.sum((X_cf_samples - X_cf_true[np.newaxis, :, :])**2, axis=2))
rmse_t = np.sqrt(np.mean(distances**2, axis=0))
return rmse_t
Common pitfalls
- Chaotic systems exhibit sensitive dependence on initial conditions; a perturbation as small as 10^-4 can cause trajectories to diverge exponentially, making exact matching impossible.
- RMSE is computed per time step t, not as a single scalar, so reporting must specify the time horizon or average over T.
- Particle filter performance heavily depends on the number of particles (N=M=200) and noise variance settings (aI, bI), which drastically alter trajectory spread.
Evidence (verbatim from paper)
We calculate the root mean squared error (RMSEt) over time to evaluate the accuracy of the sampled counterfactual trajectories compared to the projected counterfactual trajectory. This is done by computing the Euclidean distance at each time step in the d-dimensional phase space. Given d as the state dimension, T as the number of time steps, and N_cf as the number of counterfactual trajectories, the phase space distance between the deterministic counterfactual state X^cf_t and the i-th counterfactual state X^cf_i_t at time t is calculated as: d(t,i)=sqrt(sum_{k=1}^d (X^cf_i_{k,t} - X^cf_{k,t})^2) where X_t^cf is the true counterfactual state computed using the deterministic ODE and X_t^cf_i the i-th counterfactual generated at time t, using equation (5). Here X_t^cf=[X_1,t^cf, X_2,t^cf, ..., X_d,t^cf] and X_t^cf_i=[X_1,t^cf_i, X_2,t^cf_i, ..., X_d,t^cf_i]. The RMSE over all counterfactual trajectories at time t is then computed as: RMSE_t = sqrt(1/N_cf sum_{i=1}^{N_cf} d(t,i)^2)
Citation
@misc{aalaila2025counterfactual,
title={When Counterfactual Reasoning Fails: Chaos and Real-World Complexity},
author={Aalaila et al. (2025)},
year={2025},
note={arXiv:2503.23820}
}
- arXiv: 2503.23820