Tabular Predictive Eval

tabular-predictive-eval

Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science — Yang et al. (2024) (arXiv:2403.20208, 2024)

What this evaluates

Evaluates large language models on predictive tabular tasks, including classification, regression, and missing value imputation. It probes the model's ability to reason over structured data, handle mixed numerical and textual features, and perform few-shot or long-context learning on tables.

Datasets

• Kaggle (Classification & Regression) — total ?; splits: test (-1)
• Tabular Benchmark (Grinsztajn et al., 2022) — total ?; splits: test (-1); HF inria-soda/tabular-benchmark

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the model's discriminative capacity across classification thresholds.
• R^2 — range: (-∞, 1]
  • Coefficient of determination, calculated as 1 - (SS_res / SS_tot). Evaluates regression fit and variance explained.
• ROUGE-L — range: [0, 1]
  • F1-score of the longest common subsequence between predicted and ground-truth text. Used for missing value imputation where values are treated as text.

Input / output format

Input: Tabular data formatted as text sequences (e.g., 'column-name-0 is cell-value-0, column-name-1 is cell-value-1...'). Numerical values are standardized to 5 decimal places. For few-shot/long-context evaluation, k-nearest training examples are prepended as context.

Output: Predicted class label (classification), predicted numerical value (regression), or imputed cell value (text).

Scoring recipe

def compute_metrics(y_true, y_pred, task_type):
    if task_type == 'classification':
        return roc_auc_score(y_true, y_pred)
    elif task_type == 'regression':
        ss_res = np.sum((y_true - y_pred) ** 2)
        ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)
        return 1 - (ss_res / ss_tot)
    elif task_type == 'imputation':
        return rouge_score(targets=y_true, predictions=y_pred, rouge_types=['rougeL'])

Common pitfalls

• Failing to standardize numerical values to 5 decimal places before tokenization, leading to excessively long tokens and degraded performance.
• Using raw table formats instead of the specified text conversion for few-shot/long-context learning, which significantly reduces accuracy.
• Including 'easy' tasks from the public benchmark without filtering, which inflates performance metrics and misrepresents model capability.

Evidence (verbatim from paper)

To assess the model’s discriminative capacity and its effectiveness in distinguishing among different classes in classification tasks, we employ the ROC-AUC metric. For regression tasks, we utilize the coefficient of determination, $R^{2}$, as the evaluative metric. Additionally, to evaluate the model’s proficiency in predicting missing values, both textual and numerical, we consider these values as text and apply the ROUGE-L metric

Citation

@misc{yang2024unleashing,
  title={Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2403.20208}
}

• arXiv: 2403.20208