Carte Tabular Eval

carte-tabular-eval

CARTE: Pretraining and Transfer for Tabular Learning — Kim et al. (2024) (arXiv:2402.16785, 2024)

What this evaluates

Evaluates a graph-based neural architecture for tabular learning on single and multiple tables, testing its ability to handle mixed numerical and categorical features without requiring schema or entity matching.

Datasets

• TabLLM datasets — total ?; splits: train (-1), test (-1); repo https://github.com/soda-inria/carte
• Entity Matching datasets — total ?; splits: train (-1), test (-1); repo https://github.com/soda-inria/carte

Metrics

• performance (primary) — range: [0, 1]
  • Standard classification accuracy (fraction of correct predictions) for classification tasks, and R² or RMSE for regression tasks. Reported values in tables are 0–1, indicating accuracy or AUC.

Input / output format

Input: Tabular data containing numerical and categorical features, converted into a graph representation with string embeddings for entries and column names.

Output: Predicted class labels or regression values.

Scoring recipe

def compute_metric(predictions, gold):
    if task == 'classification':
        return (predictions == gold).mean()
    else:
        return r2_score(gold, predictions)
# Results reported as mean ± std over multiple train/test splits or random seeds.

Common pitfalls

• Evaluations use few-shot settings with varying training sizes (32 to 1024 samples), not fixed standard splits.
• Performance is averaged over multiple runs; reporting only a single run will not match the paper's results.
• Schema-matching is explicitly shown to be unnecessary for CARTE's multi-table transfer.

Evidence (verbatim from paper)

Table 5 shows the dataset specifications and detailed results on performance comparison between CARTE and baselines presented in Hegselmann et al. (2023). The datasets generally contain high fraction of numerical features (four datasets) or categorical columns with low cardinality (eight datasets).

Citation

@misc{kim2024carte,
  title={CARTE: Pretraining and Transfer for Tabular Learning},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2402.16785}
}

• arXiv: 2402.16785