Pulsnar Alpha Estimation Eval

pulsnar-alpha-estimation-eval

Positive Unlabeled Learning Selected Not At Random (PULSNAR): class proportion estimation when the SCAR assumption does not hold — Praveen Kumar et al. (2023) (arXiv:2303.08269, 2023)

What this evaluates

This benchmark evaluates the ability of Positive Unlabeled (PU) learning algorithms to accurately estimate the true proportion of positive examples ($\alpha$) within an unlabeled dataset, particularly when selection bias violates the SCAR assumption. It also probes the robustness of downstream classification performance and probability calibration under varying degrees of class imbalance and structured selection bias.

Datasets

• Synthetic SCAR — total 8000; splits: positive_set (2000), unlabeled_set (6000)
• Synthetic SNAR — total 8000; splits: positive_set (2000), unlabeled_set (6000)
• UCI Bank — total 45211; splits: positive_set (5289), unlabeled_set (39922)
• KDD Cup 2004 Particle Physics — total 50000; splits: positive_set (24861), unlabeled_set (25139)
• UCI Statlog (Shuttle) — total 43500; splits: positive_set (9392), unlabeled_set (34108)
• UCI Firewall — total 65532; splits: positive_set (27892), unlabeled_set (37640)

Metrics

• alpha_estimation (primary) — range: [0, 1]
  • Absolute error between the estimated positive proportion $\hat{\alpha}$ and the ground-truth proportion $\alpha_{true}$ in the unlabeled set. Lower values indicate better estimation.
• classification_performance_metrics — range: [0, 1]
  • Six standard classification metrics (e.g., accuracy, F1-score, AUC) computed on predicted labels or probabilities for the unlabeled set.
• probability_calibration — range: [0, 1]
  • Measures the discrepancy between predicted probabilities and actual positive frequencies (e.g., Expected Calibration Error or Brier score).

Input / output format

Input: Feature matrix X containing labeled positives (X_p) and unlabeled instances (X_u). Labels y are provided only for X_p; y_u is unknown during evaluation but known for ground-truth proportion calculation.

Output: Estimated positive proportion $\hat{\alpha}$ for the unlabeled set, along with predicted class probabilities or labels for each instance in X_u.

Scoring recipe

def compute_alpha_error(est_alpha, true_alpha):
    return abs(est_alpha - true_alpha)

def evaluate_pipeline(X_p, X_u, y_p, true_alpha, n_seeds=40):
    errors = []
    for seed in range(n_seeds):
        probs_p, probs_u = train_and_predict(X_p, X_u, y_p, seed)
        est_alpha = estimate_proportion(probs_p, probs_u)
        errors.append(compute_alpha_error(est_alpha, true_alpha))
    return sum(errors) / len(errors)

Common pitfalls

• Treating unlabeled instances as true negatives during base classifier training without adjusting for class imbalance (e.g., via scale_pos_weight), which distorts probability outputs.
• Assuming SCAR (random selection) when data exhibits SNAR (structured selection bias), causing standard PU estimators to severely underestimate alpha.
• Reporting point estimates without confidence intervals or averaging over multiple random seeds, as alpha estimation is highly sensitive to initialization and data splits.

Evidence (verbatim from paper)

We evaluated our proposed PU learning algorithms in terms of $\alpha$ estimates, six classification performance metrics, and probability calibration.

Citation

@misc{kumar2023pulsnar,
  title={Positive Unlabeled Learning Selected Not At Random (PULSNAR): class proportion estimation when the SCAR assumption does not hold},
  author={Praveen Kumar et al. (2023)},
  year={2023},
  note={arXiv:2303.08269}
}

• arXiv: 2303.08269