Cde Regression Eval

cde-regression-eval

Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression — Izbicki et al. (2026) (arXiv:2603.26611, 2026)

What this evaluates

Evaluates tabular foundation models and traditional baselines on conditional density estimation for regression tasks. It probes density accuracy, probabilistic calibration, prediction sharpness, and computational efficiency across varying training sample sizes and diverse real-world domains.

Datasets

• OpenML & SDSS DR18 regression datasets — total ?; splits: train (-1), test (-1)

Metrics

• CDE loss (primary) — range: other
  • A proper scoring rule for conditional density estimation: $L(\hat{f}) = \int\int\hat{f}(y|x)^2 dy dP_X(x) - 2\mathbb{E}_{(X,Y)}[\hat{f}(Y|X)]$. Estimated on held-out test data; values are typically negative, with more negative indicating better density estimates.
• Mean test log-likelihood — range: other
  • The cross-entropy of the predicted conditional density against observed targets. Higher values indicate better fit.
• CRPS — range: other
  • Continuous Ranked Probability Score, a proper scoring rule measuring the difference between the predictive CDF and the step function of the observed target. Lower values indicate better calibration and sharpness.
• KS statistic of PIT — range: [0, 1]
  • Kolmogorov-Smirnov statistic computed on Probability Integral Transform values $U=\hat{F}(Y|X)$. Tests if $U$ is uniformly distributed on [0,1] to assess probabilistic calibration.
• 90% predictive interval coverage — range: percent
  • Empirical fraction of test targets falling within the model's predicted 90% predictive interval. Values closer to 0.90 indicate better calibration.
• Wall-clock time — range: other
  • Total fit-and-predict computation time measured in seconds.

Input / output format

Input: Tabular covariates X (numeric and categorical, one-hot encoded) and continuous target Y. Models receive training splits of size n ∈ {50, 500, 1000, 5000, 10000, 20000} and predict conditional distributions for held-out test points.

Output: Predictive conditional density $\hat{f}(y|x)$, cumulative distribution function $\hat{F}(y|x)$, or predictive quantiles.

Scoring recipe

def compute_cde_loss(pred_density, X_test, Y_test):
    # Evaluate predicted density at observed Y_test for each X_test
    f_yx = pred_density(Y_test, X_test)
    # Approximate E[f(Y|X)^2] via sample mean of squared densities
    # Approximate E[f(Y|X)] via sample mean of densities
    loss = np.mean(f_yx**2) - 2 * np.mean(f_yx)
    return loss  # More negative values indicate better density estimates

Common pitfalls

• CDE loss values are typically negative on real data; readers often mistakenly interpret higher (less negative) values as better performance.
• Calibration is assessed via PIT uniformity (KS test) and 90% interval coverage, not standard classification calibration metrics like reliability diagrams.
• Statistical significance testing uses a conservative Welch t-test that ignores positive correlation from shared test sets, plus Holm-Bonferroni correction across all comparisons.

Evidence (verbatim from paper)

Our main density-specific metric is the CDE loss (Izbicki and Lee, 2016; Schmidt et al., 2020), $L(\hat{f})=\int\int\hat{f}(y\mid x)^{2},dy,dP_{X}(x)-2,\mathbb{E}_{(X,Y)}[\hat{f}(Y\mid X)].$ This is a proper scoring rule, minimized in expectation by the true conditional density. Note that CDE loss values are typically negative on real data, with more negative values indicating better density estimates. We estimate it on held-out test data. We also report mean test log-likelihood (i.e., the cross-entropy) and the continuous ranked probability score (CRPS) (Gneiting and Raftery, 2007), which are also proper scoring rules for CDE.

Citation

@misc{izbicki2026benchmarking,
  title={Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression},
  author={Izbicki et al. (2026)},
  year={2026},
  note={arXiv:2603.26611}
}

• arXiv: 2603.26611