Signal Quality Auditing Eval

signal-quality-auditing-eval

Signal Quality Auditing for Time-series Data — Gao et al. (2024) (arXiv:2402.00803, 2024)

What this evaluates

Evaluates the effectiveness of various Signal Quality Indices (SQIs) and machine learning models for classifying noisy vs. clean ECG signals, detecting outliers, and denoising time-series data.

Datasets

• Physionet 2011 ECG Challenge (PICC) Set A — total 1996; splits: train (998), test (998)
• MIT-BIH Arrhythmia & NSTDB — total 18816; splits: train (9408), test (9408)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• Accuracy — range: [0, 1]
  • Fraction of correctly classified samples out of the total number of samples.
• Mean Squared Error (MSE) (primary) — range: other
  • Average of the squared differences between the predicted denoised ECG values and the actual clean ECG ground truth values.

Input / output format

Input: 10-second 12-lead ECG recordings sampled at 500 Hz, or 512-sample windows of ECG signals.

Output: Binary label (0=not noisy, 1=noisy) for classification/outlier tasks; continuous denoised ECG waveform for denoising task.

Scoring recipe

def evaluate(y_true, y_pred, y_clean=None):
    acc = np.mean(y_true == y_pred)
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    auc = np.trapz(tpr, fpr)
    mse = np.mean((y_clean - y_pred) ** 2) if y_clean is not None else None
    return {'accuracy': acc, 'AUC': auc, 'MSE': mse}

Common pitfalls

• Data leakage across subjects if not using subject-based 5-fold cross-validation.
• Using PhysioNet Set B instead of Set A, which breaks reproducibility with the reported results.
• Comparing denoising MSE across different noise types/SNR levels without normalizing or specifying the exact noise sources (EM, BW, MA, GN).

Evidence (verbatim from paper)

We used 5-fold group split cross-validation on the subjects in Set-A of the dataset, so subject data do not overlap in training and testing in any iteration. That yieldED 1996 ECG time-series for training and testing (50% split), each 10 seconds long, recorded at 500 Hz. These raw time-series signals are then featurized via the SQIs mentioned above. This yieldED a straightforward binary classification: 0=not noisy, 1=noisy. A Random Forest classifier was further trained on these SQI attributes. Table 3: Table of AUCs and Accuracies (with standard deviations) of 8 methods on subject-based 5-fold cross validation and Random Forest with only single-lead features.

Citation

@misc{gao2024signal,
  title={Signal Quality Auditing for Time-series Data},
  author={Gao et al. (2024)},
  year={2024},
  note={arXiv:2402.00803}
}

• arXiv: 2402.00803