Comp Eval

comp-eval

Contrastive Mixture of Posteriors for Counterfactual Inference, Data Integration and Fairness — Foster et al. (2021) (arXiv:2106.08161, 2021)

What this evaluates

Evaluates a model's ability to align latent representations across different conditions (e.g., batch effects, treatment, demographic attributes) while preserving task-relevant information. It measures local mixing quality using nearest-neighbour and silhouette metrics, and assesses predictive utility via classification accuracy on held-out labels.

Datasets

• Tumour / Cell Line — total ?; splits: (unstated)
• Stimulated / untreated single-cell PBMCs — total ?; splits: (unstated)
• Single-cell RNA-seq data integration (PBMCs) — total ?; splits: (unstated)
• UCI Adult Income — total ?; splits: (unstated)

Metrics

• kBET (primary) — range: [0, 1]
  • k-Batch Effect Test (kBET) measures the rejection rate of the null hypothesis that the proportion of samples from a given batch in the k-nearest neighbours matches the global batch proportion. A low value close to zero indicates good local mixing.
• Silhouette Coefficient (s) — range: other
  • Local Silhouette Coefficient computed over k-nearest neighbours in the latent space. A low value close to zero indicates good local mixing of sample representations.
• m-kBET — range: [0, 1]
  • Mean kBET calculated by computing kBET separately for every fixed subpopulation (e.g., cell type) and averaging the results. Penalizes global alignment that mixes different subpopulations.
• mean Silhouette Coefficient (\tilde{s}) — range: other
  • Mean Silhouette Coefficient calculated by computing the local Silhouette Coefficient separately for every fixed subpopulation and averaging the results.
• Accuracy — range: [0, 1]
  • Standard classification accuracy on held-out labels (e.g., cancer type, income, gender) predicted from the learned latent representations.

Input / output format

Input: Gene expression profiles (bulk or single-cell) or tabular census data, paired with condition labels (e.g., tumour/cell-line, stimulated/untreated, library protocol, gender/income).

Output: Latent representations (z) and/or reconstructed gene expression profiles. For classification tasks, predicted class labels.

Scoring recipe

def compute_kBET(latents, batch_labels, k=100, alpha=0.01):
    # Compute k-NN in latent space
    # For each sample, calculate proportion of neighbors from same batch
    # Return rejection rate (low is good)
    pass

def compute_mean_kBET(latents, batch_labels, subpop_labels):
    scores = [compute_kBET(latents[sp], batch_labels[sp]) for sp in unique(subpop_labels)]
    return mean(scores)

def compute_accuracy(predictions, gold_labels):
    return sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)

Common pitfalls

• Low values for kBET and Silhouette Coefficient indicate good mixing, which is counterintuitive compared to standard accuracy metrics.
• The mean variants (m-kBET, \tilde{s}) specifically penalize models that achieve global alignment but erroneously cluster different subpopulations together.
• Evaluation metrics are sometimes computed on 2D UMAP projections rather than the full high-dimensional latent space.

Evidence (verbatim from paper)

To benchmark CoMP on the first objective, we use the following pair of k nearest-neighbour metrics: kBET, the metric used to evaluate batch correction methods in biology, and a local Silhouette Coefficient s_{k,c}. In both cases a low value close to zero indicates good local mixing of sample representations.

Citation

@misc{foster2021comp,
  title={Contrastive Mixture of Posteriors for Counterfactual Inference, Data Integration and Fairness},
  author={Foster et al. (2021)},
  year={2021},
  note={arXiv:2106.08161}
}

• arXiv: 2106.08161