highly-variable-gene-selection
Summary
Identify and retain genes with high variance across single cells to reduce dimensionality while preserving biological signal. This preprocessing step filters a typically genome-scale gene set (e.g., ~20,000 genes in human) down to a smaller, more informative subset (typically hundreds to thousands) before downstream analysis.
When to use
Apply this skill after normalization (normalize_total, log1p transformation) and before PCA or other dimensionality reduction on raw or near-raw single-cell gene expression matrices. Use it when working with whole-genome expression data and you need to reduce computational burden while retaining genes most likely to distinguish cell types or biological states.
When NOT to use
- Input is already a pre-filtered feature table (e.g., from a prior study); redundant filtering may lose known markers.
- Analysis requires discovery of rare or lowly-expressed genes (e.g., transcription factors, cell-type-specific markers with low absolute expression); variance-based selection may exclude them.
- Working with targeted or amplicon sequencing data pre-selected for specific genes of interest; apply only if you intend to expand the feature set.
Inputs
- AnnData object with normalized, log-transformed gene expression matrix (n_obs × n_vars)
- Cell-level metadata (obs) and gene-level metadata (var)
Outputs
- AnnData object with highly_variable boolean column in .var
- Subset AnnData object containing only highly variable genes
- Variance metrics and thresholds stored in .var annotations
How to apply
Use Scanpy's pp.highly_variable_genes() function to identify genes with high variance across the cell population. The function computes variance metrics (typically within-batch or across-batch) and ranks genes; genes above a specified variance threshold are retained. This step is applied to a normalized, log-transformed AnnData object (obs × genes matrix). The selected genes are then used for all subsequent analysis (PCA, neighbor graph, clustering, embedding); genes failing to meet the variance threshold are excluded to reduce noise and improve computational efficiency without sacrificing biological interpretability.
Related tools
- Scanpy (Primary library providing pp.highly_variable_genes() function for variance-based gene filtering) — https://github.com/scverse/scanpy
- anndata (Data structure (AnnData) for storing and manipulating gene expression matrices with metadata) — https://github.com/scverse/anndata
- Python (Programming language and runtime for executing Scanpy preprocessing functions)
Examples
import scanpy as sc
adata = sc.read_h5ad('pbmc3k_normalized.h5ad')
sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5)
adata_hvg = adata[:, adata.var.highly_variable]
sc.pp.pca(adata_hvg)
Evaluation signals
- AnnData.var contains a 'highly_variable' boolean column with True/False for each gene
- Number of retained genes is substantially smaller than input (typically <20% of original gene count) but sufficient for downstream analysis
- Retained genes show high variance across the cell population (inspect .var['dispersions'] or equivalent metric)
- PCA and clustering on highly variable genes produce interpretable, biologically meaningful cell clusters consistent with known biology
- Highly variable genes include known marker genes for the expected cell types in the dataset
Limitations
- Variance-based selection assumes noise is uniformly distributed and that high-variance genes are biologically informative; this may fail if batch effects dominate variance structure.
- Rare cell types or transient cell states with low variance within-type but high between-type variance may be missed.
- Genes with uniform or bimodal expression (e.g., on/off markers) may have moderate variance and pass filtering; additional filtering (e.g., by mean expression) may be needed.
- The method is sensitive to normalization choice; improper normalization before variance calculation will bias gene selection.
Evidence
- [other] Identify highly variable genes using pp.highly_variable_genes to reduce dimensionality.: "Identify highly variable genes using pp.highly_variable_genes to reduce dimensionality."
- [intro] Scanpy is a scalable toolkit for analyzing single-cell gene expression data built jointly with anndata.: "Scanpy is a scalable toolkit for analyzing single-cell gene expression data built jointly with anndata."
- [intro] Scanpy includes preprocessing, visualization, clustering, trajectory inference and differential expression testing capabilities: "It includes preprocessing, visualization, clustering, trajectory inference and differential expression testing"
1---2name: highly-variable-gene-selection3description: Use when after normalization (normalize_total, log1p transformation) and before PCA or other dimensionality reduction on raw or near-raw single-cell gene expression matrices.4license: CC-BY-4.05---67# highly-variable-gene-selection89## Summary1011Identify and retain genes with high variance across single cells to reduce dimensionality while preserving biological signal. This preprocessing step filters a typically genome-scale gene set (e.g., ~20,000 genes in human) down to a smaller, more informative subset (typically hundreds to thousands) before downstream analysis.1213## When to use1415Apply this skill after normalization (normalize_total, log1p transformation) and before PCA or other dimensionality reduction on raw or near-raw single-cell gene expression matrices. Use it when working with whole-genome expression data and you need to reduce computational burden while retaining genes most likely to distinguish cell types or biological states.1617## When NOT to use1819- Input is already a pre-filtered feature table (e.g., from a prior study); redundant filtering may lose known markers.20- Analysis requires discovery of rare or lowly-expressed genes (e.g., transcription factors, cell-type-specific markers with low absolute expression); variance-based selection may exclude them.21- Working with targeted or amplicon sequencing data pre-selected for specific genes of interest; apply only if you intend to expand the feature set.2223## Inputs2425- AnnData object with normalized, log-transformed gene expression matrix (n_obs × n_vars)26- Cell-level metadata (obs) and gene-level metadata (var)2728## Outputs2930- AnnData object with highly_variable boolean column in .var31- Subset AnnData object containing only highly variable genes32- Variance metrics and thresholds stored in .var annotations3334## How to apply3536Use Scanpy's pp.highly_variable_genes() function to identify genes with high variance across the cell population. The function computes variance metrics (typically within-batch or across-batch) and ranks genes; genes above a specified variance threshold are retained. This step is applied to a normalized, log-transformed AnnData object (obs × genes matrix). The selected genes are then used for all subsequent analysis (PCA, neighbor graph, clustering, embedding); genes failing to meet the variance threshold are excluded to reduce noise and improve computational efficiency without sacrificing biological interpretability.3738## Related tools3940- **Scanpy** (Primary library providing pp.highly_variable_genes() function for variance-based gene filtering) — https://github.com/scverse/scanpy41- **anndata** (Data structure (AnnData) for storing and manipulating gene expression matrices with metadata) — https://github.com/scverse/anndata42- **Python** (Programming language and runtime for executing Scanpy preprocessing functions)4344## Examples4546```47import scanpy as sc48adata = sc.read_h5ad('pbmc3k_normalized.h5ad')49sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5)50adata_hvg = adata[:, adata.var.highly_variable]51sc.pp.pca(adata_hvg)52```5354## Evaluation signals5556- AnnData.var contains a 'highly_variable' boolean column with True/False for each gene57- Number of retained genes is substantially smaller than input (typically <20% of original gene count) but sufficient for downstream analysis58- Retained genes show high variance across the cell population (inspect .var['dispersions'] or equivalent metric)59- PCA and clustering on highly variable genes produce interpretable, biologically meaningful cell clusters consistent with known biology60- Highly variable genes include known marker genes for the expected cell types in the dataset6162## Limitations6364- Variance-based selection assumes noise is uniformly distributed and that high-variance genes are biologically informative; this may fail if batch effects dominate variance structure.65- Rare cell types or transient cell states with low variance within-type but high between-type variance may be missed.66- Genes with uniform or bimodal expression (e.g., on/off markers) may have moderate variance and pass filtering; additional filtering (e.g., by mean expression) may be needed.67- The method is sensitive to normalization choice; improper normalization before variance calculation will bias gene selection.6869## Evidence7071- [other] Identify highly variable genes using pp.highly_variable_genes to reduce dimensionality.: "Identify highly variable genes using pp.highly_variable_genes to reduce dimensionality."72- [intro] Scanpy is a scalable toolkit for analyzing single-cell gene expression data built jointly with anndata.: "Scanpy is a scalable toolkit for analyzing single-cell gene expression data built jointly with anndata."73- [intro] Scanpy includes preprocessing, visualization, clustering, trajectory inference and differential expression testing capabilities: "It includes preprocessing, visualization, clustering, trajectory inference and differential expression testing"
HolobiomicsLab