Privacybench Eval

privacybench-eval

PrivacyBench: Privacy Isn't Free in Hybrid Privacy-Preserving Vision Systems — Obiefuna et al. (2026) (arXiv:2602.18900, 2026)

What this evaluates

Evaluates the trade-offs between privacy preservation, model utility, and computational/energy costs in hybrid privacy-preserving vision systems. It probes how combining federated learning with differential privacy or secure multi-party computation affects convergence, classification accuracy, and resource consumption across different neural architectures.

Datasets

• Alzheimer MRI Classification — total ?; splits: train (-1), val (-1)
• ISIC Skin Lesion Classification — total ?; splits: train (-1), val (-1)

Metrics

• MCC (primary) — range: [-1, 1]
  • Matthews Correlation Coefficient measures the quality of binary/multiclass classifications. It returns a value between -1 and +1, where +1 is perfect prediction, 0 is random prediction, and -1 is inverse prediction.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.

Input / output format

Input: Preprocessed MRI brain scans or dermoscopic images fed into ResNet18 or ViT-Base architectures.

Output: Class predictions (4 classes for Alzheimer MRI, 8 classes for ISIC Skin Lesion) with associated confidence scores or logits.

Scoring recipe

def evaluate(y_true, y_pred):
    accuracy = np.mean(y_true == y_pred)
    mcc = matthews_corrcoef(y_true, y_pred)  # Multiclass-aware per sklearn convention
    return {'accuracy': accuracy, 'mcc': mcc}

Common pitfalls

• Assuming privacy techniques combine additively; the benchmark shows hybrid combinations (e.g., FL+DP) cause non-additive convergence failures and compounding overhead.
• Evaluating privacy methods in isolation rather than testing hybrid configurations (FL+DP, FL+SMPC) that reflect real-world deployment constraints.
• Focusing solely on accuracy while ignoring critical resource metrics like energy consumption (kWh) and memory utilization, which determine deployment feasibility.

Evidence (verbatim from paper)

Classification Performance: Standard metrics including accuracy, F1-score, Matthews Correlation Coefficient (MCC), precision, recall, and ROC-AUC, with particular emphasis on MCC for robust evaluation of privacy technique impacts on learning quality*[[12] over f1 score and accuracy in binary classification evaluation")]*.

Citation

@misc{obiefuna2026privacybench,
  title={PrivacyBench: Privacy Isn't Free in Hybrid Privacy-Preserving Vision Systems},
  author={Obiefuna et al. (2026)},
  year={2026},
  note={arXiv:2602.18900}
}

• arXiv: 2602.18900