Tabular Transfer Eval

tabular-transfer-eval

Large Scale Transfer Learning for Tabular Data via Language Modeling — Gardner et al. (2024) (arXiv:2406.12031, 2024)

What this evaluates

Evaluates zero-shot and few-shot transfer learning capabilities of a language model on diverse tabular prediction tasks. It probes the model's ability to generalize across unseen datasets without fine-tuning, leveraging serialized row data and column headers to predict categorical or regression targets.

Datasets

• UniPredict Benchmark — total ?; splits: test (128)
• Grinsztajn Benchmark — total ?; splits: test (128)
• AutoML Multimodal Benchmark (AMLB) — total ?; splits: test (128)
• OpenML CC-18 Benchmark — total ?; splits: test (128)
• OpenML CTR-23 Benchmark — total ?; splits: test (128)

Metrics

• open-vocabulary accuracy (primary) — range: [0, 1]
  • The model generates tokens until the <|endofcompletion|> token. Accuracy is calculated as the fraction of instances where the generated text exactly matches the correct completion, including the terminating token.

Input / output format

Input: Serialized key-value pairs of a single row, along with the set of column names and possible label values. For few-shot evaluation, k serialized labeled examples (shots) are prepended to the prompt.

Output: An arbitrary sequence of tokens generated by the language model, terminated by the <|endofcompletion|> token.

Scoring recipe

correct = 0
total = 0
for instance in test_set:
    prompt = serialize(instance)
    generated = model.generate(prompt, stop_token="<|endofcompletion|>")
    if generated == instance.gold_completion:
        correct += 1
    total += 1
return correct / total

Common pitfalls

• Few-shot evaluation here strictly means inference-time prompting without gradient updates or fine-tuning; do not confuse with methods that train on test examples.
• Evaluation uses open-vocabulary exact match, which is stricter than closed-vocabulary probability assignment; models cannot simply pick the highest probability label from a fixed set.
• Results for k > 32 shots are only reported for datasets that fit within the 8192-token context window, which biases the sample toward smaller feature sets.

Evidence (verbatim from paper)

Here, we use open-vocabulary (or "open-ended") accuracy [1, 8] as the main evaluation metric for our model. In this setup, once the model is prompted with a serialized example, it is allowed to generate an arbitrary sequence of tokens. Once it produces the <|endofcompletion|> token, the generated text is then directly compared to the correct completion. Only an exact match, including the terminating <|endofcompletion|> token, is counted as accurate.

Citation

@misc{gardner2024tabula,
  title={Large Scale Transfer Learning for Tabular Data via Language Modeling},
  author={Gardner et al. (2024)},
  year={2024},
  note={arXiv:2406.12031}
}

• arXiv: 2406.12031