Missing Indicator Eval

missing-indicator-eval

The Missing Indicator Method: From Low to High Dimensions — Van Ness et al. (2022) (arXiv:2211.09259, 2022)

What this evaluates

Evaluates the performance of missing data preprocessing strategies (mean imputation, missForest, Gaussian Copula imputation, with and without Missing Indicator Method) across linear, tree-based, and neural network models. It probes how well these methods handle informative versus uninformative missingness patterns in both low and high-dimensional tabular settings.

Datasets

• Synthetic Low-Dimensional — total 10000; splits: train (-1), test (-1); repo https://github.com/mvanness354/missing_indicator_method
• Synthetic High-Dimensional — total 10000; splits: train (-1), test (-1); repo https://github.com/mvanness354/missing_indicator_method
• OpenML (12 subsets) — total ?; splits: train (-1), test (-1); repo https://github.com/mvanness354/missing_indicator_method

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error: sqrt(mean((y_true - y_pred)^2)). Used for regression tasks.
• 1 - AUC (primary) — range: [0, 1]
  • One minus the Area Under the Receiver Operating Characteristic Curve. Used for binary classification tasks.
• 1 - accuracy (primary) — range: [0, 1]
  • One minus classification accuracy. Used for multiclass classification tasks.

Input / output format

Input: Tabular feature matrix X with missing values, where missingness is generated via a self-masking mechanism P(R_j=0|X_j) = 1/(1+exp(-λ_j X_j)). Features are standardized over observed entries.

Output: Continuous predictions for regression, or class probabilities/labels for binary and multiclass classification.

Scoring recipe

def compute_metric(y_true, y_pred, task_type):
    if task_type == 'regression':
        return np.sqrt(np.mean((y_true - y_pred) ** 2))
    elif task_type == 'binary':
        return 1 - roc_auc_score(y_true, y_pred)
    elif task_type == 'multiclass':
        return 1 - accuracy_score(y_true, y_pred)

Common pitfalls

• Assuming MIM universally improves performance; it can cause overfitting in high dimensions if uninformative indicators are retained (SMIM is required to filter them).
• Misinterpreting tree-based model results; XGBoost with mean imputation already captures missingness signals via feature binning, so MIM provides little additional benefit unless non-constant imputation is used.
• Failing to standardize features over observed entries before imputation, which breaks the theoretical equivalence of mean imputation to 0-imputation used in the experiments.

Evidence (verbatim from paper)

For performance metrics, we use RMSE for regression tasks, 1 - AUC for binary classification tasks, and 1 - accuracy for multiclass classification tasks (so in all cases lower is better).

Citation

@misc{vaness2022missing,
  title={The Missing Indicator Method: From Low to High Dimensions},
  author={Van Ness et al. (2022)},
  year={2022},
  note={arXiv:2211.09259}
}

• arXiv: 2211.09259