Scssl Bench Eval

scssl-bench-eval

scSSL-Bench: Benchmarking Self-Supervised Learning for Single-Cell Data — Ovcharenko et al. (arXiv:2506.10031, 2025)

What this evaluates

Evaluates how well self-supervised learning models learn single-cell representations for three downstream tasks: batch correction, cell type annotation, and missing modality prediction. It probes the trade-off between preserving biological variance and removing technical batch effects, as well as the ability to generalize across uni- and multi-omics data modalities.

Datasets

• PBMC-M — total ?; splits: train (-1); repo https://github.com/BoevaLab/scSSL-Bench
• BMMC — total ?; splits: train (-1); repo https://github.com/BoevaLab/scSSL-Bench
• PBMC — total ?; splits: train (-1); repo https://github.com/BoevaLab/scSSL-Bench
• Pancreas — total ?; splits: train (-1); repo https://github.com/BoevaLab/scSSL-Bench
• Immune Cell Atlas — total ?; splits: train (-1); repo https://github.com/BoevaLab/scSSL-Bench
• MCA — total ?; splits: train (-1); repo https://github.com/BoevaLab/scSSL-Bench
• Lung — total ?; splits: train (-1); repo https://github.com/BoevaLab/scSSL-Bench
• Tabula Sapiens — total ?; splits: train (-1); repo https://github.com/BoevaLab/scSSL-Bench
• HIC — total ?; splits: train (-1); repo https://github.com/BoevaLab/scSSL-Bench

Metrics

• SCIB Total (primary) — range: [0, 1]
  • Min-max scaled aggregate score combining biological conservation and batch correction metrics from the SCIB benchmark. Higher values indicate better preservation of biological signal while removing technical batch effects.
• Cell Typing Accuracy — range: [0, 1]
  • Classification accuracy of a linear probe or classifier trained on learned embeddings to predict cell type labels.
• Pearson Correlation — range: [-1, 1]
  • Pearson correlation coefficient between the original missing modality values (e.g., protein abundance) and the model's predicted values.

Input / output format

Input: Gene expression matrices (and multi-modal protein/ADT matrices) where rows are cells and columns are genes/proteins.

Output: Learned low-dimensional embeddings (typically 64 dimensions) for representation learning tasks, or predicted continuous values for missing modalities.

Scoring recipe

# 1. Batch Correction (SCIB Total)
bio = compute_bio_conservation(embeddings, cell_labels)
batch = compute_batch_correction(embeddings, batch_labels)
total = minmax_scale(bio, batch)

# 2. Cell Typing
clf = train_classifier(embeddings, cell_type_labels)
acc = evaluate_accuracy(clf, test_embeddings, test_labels)

# 3. Missing Modality
pred = model.predict_missing_modality(input_modality)
corr = pearsonr(original_modality, pred)

Common pitfalls

• Overcorrecting batch effects can erase true biological variance, leading to high batch scores but low bio conservation scores.
• Using the projection head during inference degrades performance; the protocol explicitly requires discarding it at test time.
• Assuming specialized single-cell models outperform generic SSL methods; the benchmark shows generic contrastive methods often win on multi-modal tasks.

Evidence (verbatim from paper)

Figure 3 shows the ability to predict missing protein values while given only RNA or gene expression (GEX) during inference. The model is trained on multi-omics data using CLEAR augmentations and concatenation to combine modalities. The standard deviation is close to zero, see Table H5. VICReg and SimCLR outperform other methods, including specialized single-cell frameworks. We assume that Concerto, scCLIP, and scTEL do not learn enough information about the secondary modality (protein) and its connection to the main modality (RNA) and, therefore, are not able to predict the missing modality. We evaluate scButterfly in two modes: averaging kNN, as done for other methods, and generating proteins directly from gene expression data. While the performance differences between the two approaches are insignificant, the scButterfly's Pearson correlation is unsatisfactory low compared to generic contrastive methods. High values of Pearson's correlations show that models effectively infer protein values from gene expression data.

Citation

@misc{ovcharenko2025scsslbench,
  title={scSSL-Bench: Benchmarking Self-Supervised Learning for Single-Cell Data},
  author={Ovcharenko et al.},
  year={2025},
  note={arXiv:2506.10031}
}

• arXiv: 2506.10031