Tabular Icl Eval

tabular-icl-eval

Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning — Mohamed Bouadi et al. (arXiv:2512.00181, 2025)

What this evaluates

This evaluation probes a model's ability to perform few-shot in-context learning and standard classification on high-dimensional, heterogeneous tabular data. It specifically measures how well biaxial attention and meta-learning improve performance across medical, financial, and energy domains, and how robust the model is to varying support set sizes and selection strategies.

Datasets

• TALENT — total ?; splits: train (-1), test (-1)
• OpenML-CC18 — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (ACC) (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels out of the total number of test instances.
• class-weighted F1 — range: [0, 1]
  • Macro-averaged F1 score weighted by class support to account for dataset imbalance.
• mean rank — range: other
  • Average rank of a model across all datasets within a specific domain, ranked by accuracy.

Input / output format

Input: Tabular feature matrix for a support set (k examples sampled from training, all classes represented) and a test set. For standard evaluation, the model receives the support set as in-context examples and the test set as queries.

Output: Predicted class label for each test instance.

Scoring recipe

def compute_metrics(y_true, y_pred):
    acc = np.mean(y_true == y_pred)
    f1 = f1_score(y_true, y_pred, average='weighted')
    return acc, f1

# For mean rank across datasets in a domain:
# ranks = [model_acc for model in models]
# mean_rank = np.mean(scipy.stats.rankdata(ranks, method='min'))

Common pitfalls

• Mean rank is computed per domain (Medical, Finance, Energy) rather than globally across all datasets.
• Few-shot support sets must explicitly ensure all classes are represented when sampling k examples.
• Official train/test splits from the original benchmarks are used by default, overriding any custom 80/20 splits unless explicitly stated for support quality analysis.

Evidence (verbatim from paper)

For each dataset, we report overall classification accuracy (ACC), class-weighted F1 to account for imbalance, and mean rank across datasets within each domain based on accuracy. All models use official train/test splits unless noted.

Citation

@misc{bouadi2025orionbix,
  title={Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning},
  author={Mohamed Bouadi et al.},
  year={2025},
  note={arXiv:2512.00181}
}

• arXiv: 2512.00181