Tabularfm Eval

tabularfm-eval

TabularFM: An Open Framework For Tabular Foundational Models — Quan M. Tran et al. (2024) (arXiv:2406.09837, 2024)

What this evaluates

Evaluates the cross-dataset transferability of pretrained tabular generative models by measuring how well synthesized data preserves column distributions and pairwise correlations compared to ground truth tables.

Datasets

• Kaggle — total ?; splits: train (-1), test_random (144), test_domain (248)
• GitTables — total ?; splits: train (-1), test (126)

Metrics

• column shape — range: [0, 1]
  • Measures the similarity between the marginal distributions of columns in generated data versus ground truth. Averaged across all test tables.
• column pair trends — range: [0, 1]
  • Measures the preservation of pairwise correlations or rank trends between columns in generated data versus ground truth. Averaged across all test tables.
• overall average (primary) — range: [0, 1]
  • The arithmetic mean of the averaged column shape and column pair trends scores across all test tables. Larger values indicate better generation quality.

Input / output format

Input: Tabular datasets (training and test tables) from Kaggle and GitTables.

Output: Synthesized/generated tabular data produced by the model.

Scoring recipe

shape_scores = []
trends_scores = []
for gold_table, pred_table in zip(gold_tables, predictions):
    shape_scores.append(compute_column_shape_similarity(gold_table, pred_table))
    trends_scores.append(compute_pair_trends_similarity(gold_table, pred_table))
avg_shape = sum(shape_scores) / len(shape_scores)
avg_trends = sum(trends_scores) / len(trends_scores)
overall = (avg_shape + avg_trends) / 2
return overall

Common pitfalls

• Domain-based splits yield slightly lower transferability scores than random splits, which may mislead expectations about robustness.
• Adding meta-information to pretrained models did not improve performance, indicating a challenge in selecting optimal auxiliary features.
• Text-based transformers pretrained on tabular data can underperform their non-pretrained counterparts due to distribution shift or overfitting.

Evidence (verbatim from paper)

Evaluation metrics, including column shape, column pair trends, and the overall average, were averaged across all test tables. The reported metrics (larger is better) and the standard deviation are shown in brackets.

Citation

@misc{tran2024tabularfm,
  title={TabularFM: An Open Framework For Tabular Foundational Models},
  author={Quan M. Tran et al. (2024)},
  year={2024},
  note={arXiv:2406.09837}
}

• arXiv: 2406.09837