Tabicl Tabular Eval

tabicl-tabular-eval

Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models — Wen et al. (2025) (arXiv:2502.03147, 2025)

What this evaluates

Evaluates the ability of retrieval-augmented large language models to perform in-context learning on tabular data for classification and regression tasks. It probes how well non-parametric retrieval of support instances scales with dataset size and compares against numeric-based and classic tabular baselines.

Datasets

• Held-out Tabular Benchmark — total 69; splits: test (69)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance. Ranges from 0 to 1, where 1 indicates perfect discrimination.
• NMAE (primary) — range: other
  • Mean Absolute Error normalized by the label mean: MAE / mean(y_true). Lower values indicate better regression performance. Scales relative to the target's magnitude.

Input / output format

Input: Tabular feature vectors for query instances, accompanied by a set of retrieved support instances (features and labels) provided as in-context examples in natural language/text format.

Output: Predicted class labels (for classification) or continuous target values (for regression).

Scoring recipe

def compute_metrics(y_true, y_pred, task_type):
    if task_type == 'classification':
        return roc_auc_score(y_true, y_pred)
    elif task_type == 'regression':
        mae = mean_absolute_error(y_true, y_pred)
        return mae / mean(y_true)
    else:
        raise ValueError('Unknown task type')

Common pitfalls

• Data leakage must be strictly avoided by explicitly excluding any datasets used during the base LLM's post-training phase.
• The default retrieval policy may fail on datasets with specific feature distributions, requiring retrieval engineering or alternative normalization strategies.
• LLM-based TabICL outputs text-based predictions which may differ from numeric baselines, affecting direct metric comparison without proper normalization or ensemble handling.

Evidence (verbatim from paper)

For classification tasks, we use the Area Under the Receiver Operating Characteristic curve (AUROC) as the primary evaluation metric. For regression tasks, we employ the Mean Absolute Error normalized by the label mean (NMAE).

Citation

@misc{wen2025scalable,
  title={Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models},
  author={Wen et al. (2025)},
  year={2025},
  note={arXiv:2502.03147}
}

• arXiv: 2502.03147