Pulseimpute Eval

pulseimpute-eval

PulseImpute: A Novel Benchmark Task for Pulsative Physiological Signal Imputation — Xu et al. (2022) (arXiv:2212.07514, 2022)

What this evaluates

Evaluates models on imputing missing values in pulsative physiological signals (ECG and PPG) under realistic, data-driven missingness patterns. It further assesses clinical utility by measuring downstream performance on heartbeat detection and cardiac classification tasks.

Datasets

• ECG — total ?; splits: test (-1)
• PPG — total ?; splits: test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error between the imputed signal and the ground-truth signal values.
• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall for binary heartbeat detection.
• Precision — range: [0, 1]
  • True Positives divided by the sum of True Positives and False Positives for heartbeat detection.
• Sensitivity — range: [0, 1]
  • True Positives divided by the sum of True Positives and False Negatives for heartbeat detection.
• Macro-AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve averaged across all cardiac classification label categories.

Input / output format

Input: Time-series physiological signals (ECG or PPG) with simulated missing values based on real-world mHealth patterns.

Output: Reconstructed time-series signal matching the original length, plus binary detection labels for heartbeats and diagnostic labels for cardiac classification.

Scoring recipe

# Imputation
mse = mean((y_pred - y_true) ** 2)

# Heartbeat Detection
tp = sum((y_pred == 1) & (y_true == 1))
fp = sum((y_pred == 1) & (y_true == 0))
fn = sum((y_pred == 0) & (y_true == 1))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
sens = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * sens / (prec + sens) if (prec + sens) > 0 else 0

# Cardiac Classification
macro_auc = mean([auc(y_true_cat, y_pred_cat) for cat in categories])

Common pitfalls

• Models often hallucinate physiologically plausible patterns that do not match the ground truth, especially over long missingness gaps (up to 1 minute).
• Downstream diagnostic performance degrades significantly when imputation artifacts alter critical morphological features like R-peak height.
• Traditional short-gap imputation methods perform poorly on long-term dependencies but may inadvertently fare better in diagnostic categories due to minimal signal alteration.

Evidence (verbatim from paper)

Measures are for reconstruction performance (MSE) and Heartbeat Detection accuracy (F1 Score, Precision, and Sensitivity) with 95% Confidence Intervals.

Citation

@misc{xu2022pulseimpute,
  title={PulseImpute: A Novel Benchmark Task for Pulsative Physiological Signal Imputation},
  author={Xu et al. (2022)},
  year={2022},
  note={arXiv:2212.07514}
}

• arXiv: 2212.07514