Tembed Eval

tembed-eval

Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks — Liane Vogel et al. (arXiv:2604.21696, 2026)

What this evaluates

Evaluates the quality and efficiency of tabular embedding models across four granularity levels (cell, row, column, table) and six downstream tasks including similarity search, triplet evaluation, prediction, and retrieval. It probes whether a single embedding approach can generalize universally across diverse structured data applications or if performance is highly task- and granularity-dependent.

Datasets

• TEmBed Benchmark Suite — total 69; splits: test (-1); repo https://github.com/IBM/table-representation-evals

Metrics

• task-specific metrics (primary) — range: other
  • Varies by task: ranking metrics (e.g., nDCG, Recall@K) for similarity/retrieval, and accuracy/F1/RMSE for tabular prediction. Evaluated on deterministic test cases.
• runtime — range: other
  • Wall-clock time to generate embeddings and run inference per dataset.
• resource consumption — range: other
  • Peak CPU usage and memory/GPU memory required during evaluation.

Input / output format

Input: Tabular data provided in native structured format for tabular models, or serialized text (e.g., 'col1: val1 — col2: val2' for rows, markdown tables for tables, header+value for cells/columns) for text-based models.

Output: Fixed-dimensional embedding vectors for the specified granularity level (cell, row, column, or table), or predicted labels/values for tabular prediction tasks.

Scoring recipe

def evaluate(model, task, dataset):
    if task == 'prediction':
        preds = model.predict(dataset.features, dataset.labels)
        return compute_task_metric(preds, dataset.labels)
    else:
        embs = model.encode(dataset.input)
        return compute_ranking_metric(embs, dataset.ground_truth)

Common pitfalls

• Not all embedding models support all granularity levels (cell/row/column/table); evaluation must be restricted to supported levels per model.
• Extracting embeddings from prediction-focused models (e.g., TabPFN, TabICL) requires setting placeholder labels (zeros) to avoid performance degradation, as real labels interfere with embedding estimation.
• Serialization format for text embedding models significantly impacts results; inconsistent formatting across tasks can skew comparisons.

Evidence (verbatim from paper)

Evaluation is performed using deterministic test cases, reporting common task-specific metrics. In addition to task performance, in our framework we also track the runtime of the approaches and the resource consumption, namely CPU usage and required memory and GPU memory.

Citation

@misc{vogel2026tembed,
  title={Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks},
  author={Liane Vogel et al.},
  year={2026},
  note={arXiv:2604.21696}
}

• arXiv: 2604.21696