Sccluebenc Eval

sccluebenc-eval

scCluBench: Comprehensive Benchmarking of Clustering Algorithms for Single-Cell RNA Sequencing — Ping Xu et al. (2025) (arXiv:2512.02471, 2025)

What this evaluates

Evaluates the clustering accuracy, label consistency, and biological interpretability of various single-cell RNA-seq analysis methods. It probes how well traditional, deep learning, graph-based, and foundation model algorithms recover known cell type annotations across diverse tissues and dataset sizes.

Datasets

• scCluBench (36 human & mouse scRNA-seq datasets) — total ?; splits: test (-1)

Metrics

• Accuracy (ACC) — range: [0, 1]
  • Measures the best matching between ground truth labels U and predicted assignments V using a one-to-one mapping found via the Hungarian algorithm: ACC = max_m sum_i 1(l_i = m(u_i)) / n.
• Normalized Mutual Information (NMI) (primary) — range: [0, 1]
  • Measures consistency between predicted and true labels using mutual information normalized by the mean entropy of both assignments: NMI = sum_{i,j} (|U_i ∩ V_j|/n) log(n|U_i ∩ V_j| / (|U_i||V_j|)) / mean(-sum_i |U_i|/n log(|U_i|/n), -sum_j |V_j|/n log(|V_j|/n)).
• Adjusted Rand Index (ARI) — range: [-1, 1]
  • Evaluates similarity of two assignments ignoring permutations, correcting for chance: ARI = (sum_{i,j} C(n_ij,2) - [sum_i C(a_i,2) sum_j C(b_j,2)]/C(n,2)) / ([sum_i C(a_i,2) sum_j C(b_j,2)]/2 - [sum_i C(a_i,2) sum_j C(b_j,2)]/C(n,2)).
• F1 Score (Macro-F1) — range: [0, 1]
  • Computes the F1 score for each class independently and averages them: F1_macro = (1/K) sum_{k=1}^K (2 * P_k * R_k) / (P_k + R_k).

Input / output format

Input: Single-cell RNA-seq count matrices (HDF5 format) converted to analysis-ready objects (SingleCellExperiment or Seurat), paired with ground-truth cell type annotations.

Output: Discrete cluster labels for each cell, typically constrained to match the dataset's known number of cell types (k).

Scoring recipe

def compute_metrics(predictions, ground_truth):
    mapping = hungarian_algorithm(predictions, ground_truth)
    acc = sum(1 for p, g in zip(predictions, ground_truth) if p == mapping[g]) / len(predictions)
    nmi = normalized_mutual_info_score(ground_truth, predictions)
    ari = adjusted_rand_score(ground_truth, predictions)
    f1 = f1_score(ground_truth, predictions, average='macro')
    return acc, nmi, ari, f1

Common pitfalls

• DESC and other automatic clustering models may output a different number of clusters than the ground truth, artificially lowering ACC unless corrected via marker-overlap or post-hoc mapping.
• Large datasets require subsampling (e.g., 8,000 cells) for traditional methods like SC3, which can introduce variance not present in full-dataset evaluations.
• ACC requires an optimal one-to-one label mapping (Hungarian algorithm); naive label matching will severely underestimate performance.

Evidence (verbatim from paper)

The assessment of clustering performance relies on three established metrics from the public domain: Accuracy (ACC), Normalized Mutual Information (NMI) (Strehl and Ghosh 2002), and Adjusted Rand Index (ARI) (Vinh, Epps, and Bailey 2009). Higher values of these metrics indicate better clustering performance. Given the knowledge of the ground truth class assignments $U$ and our clustering algorithm assignment $V$ on $n$ data points.

Citation

@misc{xu2025sccluebenc,
  title={scCluBench: Comprehensive Benchmarking of Clustering Algorithms for Single-Cell RNA Sequencing},
  author={Ping Xu et al. (2025)},
  year={2025},
  note={arXiv:2512.02471}
}

• arXiv: 2512.02471