Bishep Tabular Eval

bishep-tabular-eval

BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model — Chenwei Xu et al. (arXiv:2404.03830, 2024)

What this evaluates

Evaluates the performance of the BiSHop model on tabular classification and regression tasks, probing its ability to handle mixed feature types, bi-directional cellular learning, and generalized sparse modern Hopfield layers.

Datasets

• Tabular Benchmarks (Adult, Bank, Blastchar, Income, SeismicBump, Shrutime, Spambase, Qsar, Jannis, CR) — total ?; splits: test (-1)

Metrics

• AUC (%) (primary) — range: percent
  • Area under the Receiver Operating Characteristic curve, reported as a percentage.
• R2 — range: other
  • Coefficient of determination for regression tasks, measuring the proportion of variance in the dependent variable predictable from the independent variables.

Input / output format

Input: Tabular dataset rows containing mixed categorical and numerical features.

Output: Predicted class probabilities or scores for classification; continuous target values for regression.

Scoring recipe

def compute_auc(y_true, y_pred):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr) * 100

def compute_r2(y_true, y_pred):
    ss_res = np.sum((y_true - y_pred) ** 2)
    ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)
    return 1 - (ss_res / ss_tot)

Common pitfalls

• Averaging AUC scores across all datasets masks per-dataset performance variations.
• Data rotation experiments show that BiSHop's performance drops significantly when both directions and datasets are rotated, highlighting sensitivity to orientation.
• Ablation studies must use default hyperparameters for remaining components to ensure fair comparison.

Evidence (verbatim from paper)

We report the average AUC score (in %) over 10 runs in Table 18. The results indicate the superior performance of our proposed generalized sparse modern Hopfield model across datasets.

Citation

@misc{xu2024bishep,
  title={BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model},
  author={Chenwei Xu et al.},
  year={2024},
  note={arXiv:2404.03830}
}

• arXiv: 2404.03830