# Tabular Transfer Eval

> Evaluates zero-shot and few-shot transfer learning capabilities of a language model on diverse tabular prediction tasks. It probes the model's ability to generalize across unseen datasets without fine-tuning, leveraging serialized row data and column headers to predict categorical or regression targets. Use when the user wants to benchmark on UniPredict Benchmark, Grinsztajn Benchmark, AutoML Multimodal Benchmark (AMLB), OpenML CC-18 Benchmark, OpenML CTR-23 Benchmark, or asks about evaluating this task. Reports open-vocabulary accuracy.

- Skill: `qhjqhj00/tabular-transfer-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tabular-transfer-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tabular-transfer-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tabular-transfer-eval

---


# tabular-transfer-eval

> Large Scale Transfer Learning for Tabular Data via Language Modeling — Gardner et al. (2024) (arXiv:2406.12031, 2024)

## What this evaluates

Evaluates zero-shot and few-shot transfer learning capabilities of a language model on diverse tabular prediction tasks. It probes the model's ability to generalize across unseen datasets without fine-tuning, leveraging serialized row data and column headers to predict categorical or regression targets.

## Datasets

- **UniPredict Benchmark** — total ?; splits: test (128)
- **Grinsztajn Benchmark** — total ?; splits: test (128)
- **AutoML Multimodal Benchmark (AMLB)** — total ?; splits: test (128)
- **OpenML CC-18 Benchmark** — total ?; splits: test (128)
- **OpenML CTR-23 Benchmark** — total ?; splits: test (128)

## Metrics

- `open-vocabulary accuracy` **(primary)** — range: [0, 1]
  - The model generates tokens until the <|endofcompletion|> token. Accuracy is calculated as the fraction of instances where the generated text exactly matches the correct completion, including the terminating token.

## Input / output format

**Input**: Serialized key-value pairs of a single row, along with the set of column names and possible label values. For few-shot evaluation, k serialized labeled examples (shots) are prepended to the prompt.

**Output**: An arbitrary sequence of tokens generated by the language model, terminated by the <|endofcompletion|> token.

## Scoring recipe

```python
correct = 0
total = 0
for instance in test_set:
    prompt = serialize(instance)
    generated = model.generate(prompt, stop_token="<|endofcompletion|>")
    if generated == instance.gold_completion:
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Few-shot evaluation here strictly means inference-time prompting without gradient updates or fine-tuning; do not confuse with methods that train on test examples.
- Evaluation uses open-vocabulary exact match, which is stricter than closed-vocabulary probability assignment; models cannot simply pick the highest probability label from a fixed set.
- Results for k > 32 shots are only reported for datasets that fit within the 8192-token context window, which biases the sample toward smaller feature sets.

## Evidence (verbatim from paper)

> Here, we use open-vocabulary (or "open-ended") accuracy [1, 8] as the main evaluation metric for our model. In this setup, once the model is prompted with a serialized example, it is allowed to generate an arbitrary sequence of tokens. Once it produces the <|endofcompletion|> token, the generated text is then directly compared to the correct completion. Only an exact match, including the terminating <|endofcompletion|> token, is counted as accurate.

## Citation

```bibtex
@misc{gardner2024tabula,
  title={Large Scale Transfer Learning for Tabular Data via Language Modeling},
  author={Gardner et al. (2024)},
  year={2024},
  note={arXiv:2406.12031}
}
```

- arXiv: 2406.12031