Capsul Eval

capsul-eval

CAPSUL: A Comprehensive Human Protein Benchmark for Subcellular Localization — Hu et al. (2026) (arXiv:2603.18571, 2026)

What this evaluates

Evaluates the ability of protein sequence and structure models to predict the subcellular localization compartments of human proteins. It probes multi-label classification performance under severe class imbalance, testing whether models can leverage 3D structural motifs or sequence embeddings to identify fine-grained organelle targeting patterns.

Datasets

• CAPSUL — total ?; splits: test (-1); repo https://github.com/getbetter-hyccc/CAPSUL

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall). The paper reports both micro-averaged (aggregates TP/FP/FN across all classes before computing) and macro-averaged (unweighted mean of per-class F1) variants to handle multi-label classification and class imbalance.

Input / output format

Input: Protein amino acid sequence and/or 3D structural graph where nodes represent residues (typically Cα atom positions) and edges represent spatial or sequential adjacency.

Output: Multi-label prediction vector y_hat in R^m (m = number of subcellular compartments) with values in (0,1) representing predicted probabilities for each compartment.

Scoring recipe

def compute_f1(y_true, y_pred, average='micro'):
    tp = sum((y_true == 1) & (y_pred == 1))
    fp = sum((y_true == 0) & (y_pred == 1))
    fn = sum((y_true == 1) & (y_pred == 0))
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp + fn + 1e-8)
    return 2 * precision * recall / (precision + recall + 1e-8)

Common pitfalls

• Standard BCE loss optimization neglects minority classes due to severe class imbalance, leading to poor performance on underrepresented compartments like lipid droplets or centrosomes.
• Some baseline tools (e.g., DeepLoc 2.1) do not support prediction for all 18 compartments, resulting in missing values ('/') in the evaluation tables rather than zero scores.
• Multi-label predictions require thresholding continuous probability outputs to binary values before computing precision/recall, a step not explicitly detailed in the paper.

Evidence (verbatim from paper)

Given the class imbalance in each location (i.e., the proportion of proteins localized to each subcellular compartment is often small), we consider the widely used evaluation metrics in this task: Precision, Recall, and F1-score (Jiang et al., 2021; Thumuluri et al., 2022). In addition, we utilize micro-averaged and macro-averaged F1-score to evaluate the overall performance across different categories.

Citation

@misc{hu2026capsul,
  title={CAPSUL: A Comprehensive Human Protein Benchmark for Subcellular Localization},
  author={Hu et al. (2026)},
  year={2026},
  note={arXiv:2603.18571}
}

• arXiv: 2603.18571