Tabular Feature Selection Eval

tabular-feature-selection-eval

A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning — Cherepanova et al. (2023) (arXiv:2311.05877, 2023)

What this evaluates

Evaluates feature selection methods by measuring downstream neural network performance on tabular datasets containing controlled extraneous features. It probes whether selected features improve or maintain predictive accuracy for classification and reduce error for regression tasks.

Datasets

• ALOI (AL) — total ?; splits: train (-1), val (-1), test (-1)
• California Housing (CA) — total ?; splits: train (-1), val (-1), test (-1)
• Covertype (CO) — total ?; splits: train (-1), val (-1), test (-1)
• Eye Movements (EY) — total ?; splits: train (-1), val (-1), test (-1)
• Gesture (GE) — total ?; splits: train (-1), val (-1), test (-1)
• Helena (HE) — total ?; splits: train (-1), val (-1), test (-1)
• Higgs 98k (HI) — total ?; splits: train (-1), val (-1), test (-1)
• House 16K (HO) — total ?; splits: train (-1), val (-1), test (-1)
• Jannis (JA) — total ?; splits: train (-1), val (-1), test (-1)
• Otto Group Product Classification (OT) — total ?; splits: train (-1), val (-1), test (-1)
• Year (YE) — total ?; splits: train (-1), val (-1), test (-1)
• Microsoft (MI) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified instances out of total instances.
• RMSE (primary) — range: other
  • Root mean squared error between predicted and true continuous values.

Input / output format

Input: Tabular dataset containing original features plus controlled extraneous features (noise, corruption, or second-order engineered features).

Output: A selected subset of features, which are passed to a downstream model (MLP or FT-Transformer) to generate predictions. Predictions are compared against ground truth labels.

Scoring recipe

def compute_metric(predictions, gold, task_type):
    if task_type == 'classification':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'regression':
        return (sum((p - g)**2 for p, g in zip(predictions, gold)) / len(gold)) ** 0.5

Common pitfalls

• Downstream model architecture heavily influences results; MLPs are more susceptible to noise than FT-Transformers.
• Hyperparameter tuning must be performed jointly for both the feature selection method and the downstream model using validation metrics.
• Results must be averaged over 10 random model initializations (seeds) rather than reported from a single run.

Evidence (verbatim from paper)

We measure downstream model performance using accuracy for the classification tasks and RMSE for the regression tasks.

Citation

@misc{cherepanova2023tabularfeatureselection,
  title={A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning},
  author={Cherepanova et al. (2023)},
  year={2023},
  note={arXiv:2311.05877}
}

• arXiv: 2311.05877