Dp Gnn Graph Classification Eval

dp-gnn-graph-classification-eval

Differentially Private Graph Classification with GNNs — Mueller et al. (2022) (arXiv:2202.02575, 2022)

What this evaluates

This evaluation protocol assesses the utility and privacy-utility trade-off of Graph Neural Networks trained with Differentially Private Stochastic Gradient Descent (DP-SGD) on graph-level classification tasks. It probes whether formal privacy guarantees can be maintained across diverse graph structures (molecules, fingerprints, ECG signals, synthetic graphs) without severely degrading predictive performance compared to non-private baselines.

Datasets

• Synthetic — total 1000; splits: train (600), val (100), test (300)
• Fingerprints — total 1900; splits: train (-1), val (-1), test (-1)
• Molbace — total 1513; splits: train (1210), val (151), test (152)
• ECG — total 1125; splits: train (-1), val (-1), test (-1)

Metrics

• ROC AUC (primary) — range: [0, 1]
  • Compute Area Under the Receiver Operating Characteristic Curve with a micro average for multi-class datasets.
• Accuracy — range: [0, 1]
  • Rate between the true positives and all samples.
• Sensitivity — range: [0, 1]
  • Rate between the true positives and the sum of the true positives and false negatives.
• Specificity — range: [0, 1]
  • Rate between the true negatives and the sum of the true negatives and false positives.
• F1 Score — range: [0, 1]
  • Harmonic mean of the precision and recall, using a micro averaging strategy for multi-class datasets.

Input / output format

Input: Graph-structured data where each instance is a graph with node features and adjacency/edge information. For ECG, signals are sub-sampled to 512 points per channel and structured as fully connected subgraphs based on lead placement. Synthetic graphs are Erdős-Rényi with controlled node features and edge probabilities.

Output: A single class label per graph (binary or multi-class depending on the dataset).

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob=None):
    tp = sum(y_true == 1 and y_pred == 1)
    tn = sum(y_true == 0 and y_pred == 0)
    fp = sum(y_true == 0 and y_pred == 1)
    fn = sum(y_true == 1 and y_pred == 0)
    accuracy = tp / (tp + tn + fp + fn)
    sensitivity = tp / (tp + fn)
    specificity = tn / (tn + fp)
    precision = tp / (tp + fp)
    recall = sensitivity
    f1 = 2 * precision * recall / (precision + recall)
    roc_auc = compute_auc(y_true, y_prob) # micro average for multi-class
    return {'accuracy': accuracy, 'sensitivity': sensitivity, 'specificity': specificity, 'f1': f1, 'roc_auc': roc_auc}

Common pitfalls

• Batch Normalization cannot be used with DP-SGD as it averages across the batch during the forward pass, leaking information and preventing per-sample gradient computation required for privacy accounting.
• The ECG dataset is highly imbalanced (207 LBBB vs 918 normal), making Accuracy misleading; ROC AUC and F1 Score are more appropriate primary metrics for this split.
• Synthetic dataset generation requires strict control over Erdős-Rényi edge probabilities (0.2 vs 0.3) and node feature distributions to ensure reproducibility and class separability.

Evidence (verbatim from paper)

We evaluate different scores for each model: ROC AUC, Accuracy, Sensitivity, Specificity and F1 Score. Hereby sensitivity reports the rate between the true positives and the sum of the true positives and false negatives. Specificity is the rate between the true negatives and the sum of the true negatives and false positives. The ROC AUC score is the Compute Area Under the Receiver Operating Characteristic Curve with a micro average for multi-class datasets. Accuracy is the rate between the true positives and all samples and the F1 Score reports the harmonic mean of the precision and recall, also using a micro averaging strategy for multi-class datasets.

Citation

@misc{mueller2022differentially,
  title={Differentially Private Graph Classification with GNNs},
  author={Mueller et al. (2022)},
  year={2022},
  note={arXiv:2202.02575}
}

• arXiv: 2202.02575