Multimodal Tabular Automl Eval

multimodal-tabular-automl-eval

Benchmarking Multimodal AutoML for Tabular Data with Text Fields — Shi et al. (2021) (arXiv:2111.02705, 2021)

What this evaluates

Evaluates automated machine learning strategies for supervised learning on multimodal tabular datasets containing text, numeric, and categorical features. It probes how well different featurization methods, neural backbones, and ensemble aggregation techniques handle mixed data types and extract predictive signal from text fields.

Datasets

• Multimodal Tabular Benchmark (18 datasets) — total ?; splits: train/val/test (-1); repo https://github.com/sxjscience/autom1MULTIModal_benchmark

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified instances out of the total number of instances.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the ability of a binary classifier to distinguish between classes across all classification thresholds.
• R^2 — range: other
  • Coefficient of determination for regression tasks, representing the proportion of variance in the dependent variable predictable from the independent variables.

Input / output format

Input: Tabular rows containing mixed modalities: text fields, numeric features, and categorical features.

Output: Predicted class labels (for classification) or continuous values (for regression).

Scoring recipe

def score(predictions, gold, task_type):
    if task_type == 'binary_classification':
        return roc_auc_score(gold, predictions)
    elif task_type == 'multiclass_classification':
        return accuracy_score(gold, predictions)
    elif task_type == 'regression':
        return r2_score(gold, predictions)

Common pitfalls

• Off-the-shelf AutoML tools (e.g., H2O) may automatically treat text columns as categorical, effectively ignoring the text features unless explicitly featurized beforehand.
• The coefficient of determination (R^2) can be negative for poor predictions, despite the paper noting it lies in [0, 1] for reasonable predictions.
• Featurization strategy choice (e.g., N-Grams vs. pretrained Transformer embeddings) drastically alters the predictive signal available to tabular models, making cross-method comparisons highly sensitive to preprocessing decisions.

Evidence (verbatim from paper)

We evaluate regression tasks via the coefficient of determination $R^2$ , multiclass classification tasks via accuracy, and binary classification tasks via area under the ROC curve (AUC).

Citation

@misc{shi2021benchmarkingmultimodalautoml,
  title={Benchmarking Multimodal AutoML for Tabular Data with Text Fields},
  author={Shi et al. (2021)},
  year={2021},
  note={arXiv:2111.02705}
}

• arXiv: 2111.02705