Scmamba Integration Eval

scmamba-integration-eval

scMamba: A Scalable Foundation Model for Single-Cell Multi-Omics Integration Beyond Highly Variable Feature Selection — Yuan et al. (2025) (arXiv:2506.20697, 2025)

What this evaluates

Evaluates computational methods for single-cell multi-omics integration by measuring their ability to preserve biological variation, align different omics layers at the cell and single-cell levels, and enable accurate cell type annotation.

Datasets

• SHARE-seq BMMC — total ?; splits: test (-1)
• SNARE-seq — total ?; splits: test (-1)
• 10X Genomics Multiome — total ?; splits: test (-1)
• Human fetal atlas — total 377134; splits: test (-1)
• CITE-seq BMMC S1 — total ?; splits: reference (-1), query (-1)
• CITE-seq BMMC S4 — total ?; splits: reference (-1), query (-1)
• Human brain multi-omics — total ?; splits: reference (-1)
• Human brain 3k — total ?; splits: query (-1)

Metrics

• biological variation conservation score (primary) — range: [0, 1]
  • Min-max scaled average of ARI, NMI, MAP, and cASW across all compared methods.
• omics alignment score (primary) — range: [0, 1]
  • Min-max scaled average of OEMS, SAS, GC, and oASW across all compared methods.
• overall integration score (primary) — range: [0, 1]
  • Weighted sum of biological variation conservation score (0.6) and omics alignment score (0.4).
• FOSCTTM — range: [0, 1]
  • Fraction of reference samples closer to a query cell than its true matched counterpart. Lower values indicate better alignment.
• matching score — range: [0, 1]
  • Average confidence in the accuracy of cross-omics pairings for the same cell. Higher values indicate stronger alignment.
• accuracy — range: [0, 1]
  • Proportion of correctly predicted cell type labels in the annotation task.

Input / output format

Input: Raw single-cell multi-omics matrices (e.g., gene expression counts, chromatin accessibility peaks, protein abundance) with genomic coordinates for ordering. Genes/peaks are partitioned into patches and combined with 1D positional embeddings.

Output: Cell embeddings (latent vectors) for each cell, optimized via contrastive learning. For annotation tasks, categorical cell type labels.

Scoring recipe

def compute_aggregate_scores(predictions, gold_labels, omics_labels):
    bio_scores = [ARI(predictions, gold_labels), NMI(predictions, gold_labels),
                  MAP(predictions, gold_labels), cASW(predictions, gold_labels)]
    align_scores = [OEMS(predictions, omics_labels), SAS(predictions, omics_labels),
                    GC(predictions, omics_labels), oASW(predictions, omics_labels)]
    bio_cons = mean(min_max_scale(bio_scores))
    omics_align = mean(min_max_scale(align_scores))
    overall = 0.6 * bio_cons + 0.4 * omics_align
    return overall

def compute_foscttm(query_emb, ref_emb, true_match_idx):
    dists = cdist(query_emb, ref_emb)
    true_dist = dists[true_match_idx]
    return mean((dists < true_dist).astype(float))

Common pitfalls

• Aggregate scores depend on min-max scaling across the specific set of baseline methods included in the benchmark, making them sensitive to the comparison pool.
• FOSCTTM and matching scores require ground-truth cell-to-cell correspondences, limiting their use to paired multi-omics datasets.
• Performance is highly sensitive to input preprocessing; using highly variable features (HVG/HVP) instead of raw features significantly alters baseline comparisons.

Evidence (verbatim from paper)

To assess the performance of various methods on cell clustering and preservation of biological variation, we used four metrics, including adjusted rand index (ARI), normalized mutual information (NMI), mean average precision (MAP), and average silhouette width of cell types (cASW). Similarly, to assess the performance of various methods in omics alignment, we applied four metrics, including omics entropy mixing score (OEMS), seurat alignment score (SAS), graph connectivity (GC), and average silhouette width of omics (oASW).

Citation

@misc{yuan2025scmamba,
  title={scMamba: A Scalable Foundation Model for Single-Cell Multi-Omics Integration Beyond Highly Variable Feature Selection},
  author={Yuan et al. (2025)},
  year={2025},
  note={arXiv:2506.20697}
}

• arXiv: 2506.20697