Single-Cell Genomics and Expression Matrix Analysis
RULE ZERO — Check for pre-computed results FIRST
Before following any instruction below, scan the data folder for:
*_executed.ipynb → read with tu run read_executed_notebook '{"data_folder":"<path>","search":"<keyword>"}' and cite its cell outputs as the authoritative answer
- Pre-computed result files (CSV/TSV with names like
*results*, *deseq*, *enrich*, *stats*, *_simplified.csv) → read directly and report the requested value
- Canonical analysis scripts (
analysis.R, run_*.py, find_*.R, *.Rmd) → execute as-is and read the output
Only follow this skill's re-analysis recipe below if none of the above exist. Re-running from raw data produces different numbers than the published answer and is much slower (often 5-10× turn count).
Comprehensive single-cell RNA-seq analysis and expression matrix processing using scanpy, anndata, scipy, and ToolUniverse.
LOOK UP, DON'T GUESS
When uncertain about any scientific fact, SEARCH databases first (PubMed, UniProt, ChEMBL, ClinVar, etc.) rather than reasoning from memory. A database-verified answer is always more reliable than a guess.
When to Use This Skill
Apply when users:
- Have scRNA-seq data (h5ad, 10X, CSV count matrices) and want analysis
- Need scRNA-seq quality control / QC gating: deciding cell filters by
mito % (pct_counts_mt), gene/UMI counts, doublets, ambient RNA, empty droplets
- Ask about cell type identification, clustering, or annotation
- Need differential expression analysis by cell type or condition
- Want gene-expression correlation analysis (e.g., gene length vs expression by cell type)
- Ask about PCA, UMAP, t-SNE for expression data
- Need Leiden/Louvain clustering on expression matrices
- Want statistical comparisons between cell types (t-test, ANOVA, fold change)
- Ask about marker genes, batch correction, trajectory, or cell-cell communication
NOT for (use other skills instead):
- Bulk RNA-seq DESeq2 only ->
tooluniverse-rnaseq-deseq2
- Gene enrichment only ->
tooluniverse-gene-enrichment
- VCF/variant analysis ->
tooluniverse-variant-analysis
- Deep-learning integration/batch correction (scVI, scANVI, totalVI, PeakVI, MultiVI, DestVI, veloVI) ->
tooluniverse-scvi-tools
Core Principles
- Data-first - Load, inspect, validate before analysis
- AnnData-centric - All data flows through anndata objects
- Cell type awareness - Per-cell-type subsetting when needed
- Statistical rigor - Normalization, multiple testing correction, effect sizes
- Question-driven - Parse what the user is actually asking
Required Packages
import scanpy as sc, anndata as ad, pandas as pd, numpy as np
from scipy import stats
from scipy.cluster.hierarchy import linkage, fcluster
from sklearn.decomposition import PCA
from statsmodels.stats.multitest import multipletests
import gseapy as gp # enrichment
import harmonypy # batch correction (optional)
Install: pip install scanpy anndata leidenalg umap-learn harmonypy gseapy pandas numpy scipy scikit-learn statsmodels
Workflow Decision Tree
START: User question about scRNA-seq data
|
+-- FULL PIPELINE (raw counts -> annotated clusters)
| Workflow: QC -> Normalize -> HVG -> PCA -> Cluster -> Annotate -> DE
| See: references/scanpy_workflow.md
|
+-- DIFFERENTIAL EXPRESSION (per-cell-type comparison)
| Most common pattern: per-cell-type DE
| See: analysis_patterns.md "Pattern 1"
|
+-- CORRELATION ANALYSIS (gene property vs expression)
| Pattern: Gene length vs expression correlation
| See: analysis_patterns.md "Pattern 2"
|
+-- CLUSTERING & PCA (expression matrix analysis)
| See: references/clustering_guide.md
|
+-- CELL COMMUNICATION (ligand-receptor interactions)
| See: references/cell_communication.md
|
+-- TRAJECTORY ANALYSIS (pseudotime)
See: references/trajectory_analysis.md
Data format handling:
- h5ad ->
sc.read_h5ad()
- 10X ->
sc.read_10x_mtx() or sc.read_10x_h5()
- CSV/TSV ->
pd.read_csv() -> Convert to AnnData (check orientation!)
Data Loading
AnnData expects: cells/samples as rows (obs), genes as columns (var)
adata = sc.read_h5ad("data.h5ad") # h5ad already oriented
# CSV/TSV: check orientation
df = pd.read_csv("counts.csv", index_col=0)
if df.shape[0] > df.shape[1] * 5: # genes > samples by 5x => transpose
df = df.T
adata = ad.AnnData(df)
# Load metadata
meta = pd.read_csv("metadata.csv", index_col=0)
common = adata.obs_names.intersection(meta.index)
adata = adata[common].copy()
for col in meta.columns:
adata.obs[col] = meta.loc[common, col]
Quality Control and QC Gating (do this BEFORE downstream analysis)
QC gating decides which cells and genes are real before normalization,
clustering, or DE. Skipping or rushing it propagates silently: doublets become
fake "intermediate" states, ambient RNA smears markers across clusters, and
empty droplets inflate cell counts. Never report a filtered cell count without
the gates applied, and never report cutoffs you did not actually run.
HONEST EXECUTION: QC runs scanpy/AnnData via Bash/Python. If scanpy is not
installed, do NOT fabricate metrics — print the install plan and stop:
python scripts/scrna_qc.py --install-plan (exits 0, lists what is missing,
suggests pip install scanpy anndata scrublet).
Per-cell QC metrics (what to compute)
vn = adata.var_names.str.upper()
adata.var['mt'] = vn.str.startswith('MT-') # mitochondrial
adata.var['ribo'] = vn.str.startswith(('RPS', 'RPL')) # ribosomal protein
adata.var['hb'] = vn.str.contains(r'^HB[^P]', regex=True) # hemoglobin (RBC)
sc.pp.calculate_qc_metrics(
adata, qc_vars=['mt', 'ribo', 'hb'],
percent_top=None, # REQUIRED for small gene panels (<500) — else IndexError
log1p=True, inplace=True)
Key metrics in adata.obs: n_genes_by_counts, total_counts,
pct_counts_mt, pct_counts_ribo, pct_counts_hb.
WHY each metric flags a problem (reason from biology, not magic numbers)
- High
pct_counts_mt -> dying / stressed cell. A ruptured membrane lets
cytoplasmic mRNA leak out while mito transcripts stay trapped, enriching the
captured RNA for mito. Cutoff is tissue-dependent (cardiomyocytes/hepatocytes
are mito-rich at baseline — a blanket 10% would discard healthy cells).
- Low
n_genes_by_counts / total_counts -> empty droplet or debris
(only ambient RNA captured; few genes, low depth).
- Very high counts/genes -> doublet (two transcriptomes ~double depth and
diversity) — but high count alone is weak; use a doublet caller (below).
- High
pct_counts_hb -> RBC/blood contamination in solid tissue.
Choosing thresholds — distribution-aware (MAD), not hardcoded
Hardcoded cutoffs (mt<5%, n_genes<2500) are a starting point only; they
break on mito-rich tissues and on shallow vs deep libraries. Prefer a
MAD-based rule (robust to the very outliers you are removing): flag cells
nmads median-absolute-deviations from the median. Use nmads=5 on
log1p counts/genes (both tails), nmads=3 upper-only on pct_counts_mt, and
pair mito with a biological ceiling so a uniformly degraded sample doesn't pass.
Always visualize distributions first (violin + total_counts vs
pct_counts_mt scatter — dying cells sit in the low-count/high-mito corner).
Run the helper (computes metrics + MAD gating, reports per-step removals):
python scripts/scrna_qc.py data.h5ad --doublets # or --install-plan first
Doublets, ambient RNA, empty droplets (per-cell metrics miss these)
- Doublets: Scrublet (
sc.pp.scrublet, scanpy >=1.10) or scDblFinder (R).
Run per sample before merging; flag-cluster-drop (doublets form bridge
clusters). expected_doublet_rate ~0.8%/1,000 cells recovered (10x).
- Ambient RNA: cell-free "soup" mRNA in every droplet — a count-correction
step (SoupX / DecontX, R), NOT a cell filter. Per-cell QC cannot detect it;
suspect it when markers look implausibly ubiquitous.
- Empty droplets: upstream of per-cell QC. CellRanger's filtered matrix
already applies an EmptyDrops-style call; with only the raw matrix, run
EmptyDrops (DropletUtils, R) or a barcode-rank knee before per-cell QC.
QC Interpretation Table (metric -> concern -> action)
| QC metric |
Direction |
Typical concern |
Suggested action |
pct_counts_mt |
high |
Dying / stressed cell (membrane rupture) |
Filter (MAD upper + tissue-aware ceiling; raise ceiling for mito-rich tissue) |
n_genes_by_counts |
very low |
Empty droplet / debris |
Filter (min_genes ~200 + low-tail MAD) |
total_counts |
very low |
Shallow / failed capture |
Filter (low-tail MAD; check barcode-rank knee) |
n_genes_by_counts / total_counts |
very high |
Doublet (two cells in one droplet) |
Flag, run Scrublet/scDblFinder, then drop — don't hard-cap on counts alone |
predicted_doublet (Scrublet) |
True |
Multiplet |
Filter per sample; cluster-then-drop if unsure |
pct_counts_hb |
high |
RBC / blood contamination |
Filter in non-blood tissue; investigate in blood |
pct_counts_ribo |
very high/low |
Low-complexity / stressed, or cell-type signal |
Flag / investigate (ribo is cell-type-specific; rarely a hard filter) |
| markers ubiquitous across clusters |
— |
Ambient RNA contamination |
Investigate — run SoupX/DecontX, do not silently proceed |
| many cells, low median counts |
— |
Empty droplets not removed |
Investigate — apply EmptyDrops / knee, re-filter |
Full reasoning, MAD code, Scrublet/SoupX/EmptyDrops recipes, and order of
operations: references/scrna_qc.md. Inline pipeline QC:
references/scanpy_workflow.md Phase 2.
Complete Pipeline (Quick Reference)
import scanpy as sc
adata = sc.read_10x_h5("filtered_feature_bc_matrix.h5")
# QC
adata.var['mt'] = adata.var_names.str.startswith('MT-')
sc.pp.calculate_qc_metrics(adata, qc_vars=['mt'], inplace=True)
adata = adata[adata.obs['pct_counts_mt'] < 20].copy()
sc.pp.filter_cells(adata, min_genes=200)
sc.pp.filter_genes(adata, min_cells=3)
# Normalize + HVG + PCA
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
adata.raw = adata.copy()
sc.pp.highly_variable_genes(adata, n_top_genes=2000)
sc.tl.pca(adata, n_comps=50)
# Cluster + UMAP
sc.pp.neighbors(adata, n_pcs=30)
sc.tl.leiden(adata, resolution=0.5)
sc.tl.umap(adata)
# Find markers + Annotate + Per-cell-type DE
sc.tl.rank_genes_groups(adata, groupby='leiden', method='wilcoxon')
Differential Expression Decision Tree
Single-Cell DE (many cells per condition):
Use: sc.tl.rank_genes_groups(), methods: wilcoxon, t-test, logreg
Best for: Per-cell-type DE, marker gene finding
Pseudo-Bulk DE (aggregate counts by sample):
Use: R DESeq2 via `tu run run_deseq2_analysis` or Rscript (NOT pydeseq2 — gives different DEG counts)
Best for: Sample-level comparisons with replicates
Statistical Tests Only:
Use: scipy.stats (ttest_ind, f_oneway, pearsonr)
Best for: Correlation, ANOVA, t-tests on summaries
Statistical Tests (Quick Reference)
from scipy import stats
from statsmodels.stats.multitest import multipletests
# Pearson/Spearman correlation
r, p = stats.pearsonr(gene_lengths, mean_expression)
# Welch's t-test
t_stat, p_val = stats.ttest_ind(group1, group2, equal_var=False)
# ANOVA
f_stat, p_val = stats.f_oneway(group1, group2, group3)
# Multiple testing correction (BH)
reject, pvals_adj, _, _ = multipletests(pvals, method='fdr_bh')
Batch Correction (Harmony)
import harmonypy
sc.tl.pca(adata, n_comps=50)
ho = harmonypy.run_harmony(adata.obsm['X_pca'][:, :30], adata.obs, 'batch', random_state=0)
adata.obsm['X_pca_harmony'] = ho.Z_corr.T
sc.pp.neighbors(adata, use_rep='X_pca_harmony')
sc.tl.leiden(adata, resolution=0.5)
sc.tl.umap(adata)
ToolUniverse Integration
Data Discovery (before analysis)
- CxGDisc_search_datasets: Search CELLxGENE Discover for scRNA-seq datasets by disease, tissue, organism. Use broad disease terms (e.g., "breast cancer" not "triple-negative").
- GEO_search_rnaseq_datasets / geo_search_datasets: Search GEO for scRNA-seq studies
- NCBI_SRA_search_runs: Search SRA for sequencing runs (query="single cell RNA-seq [disease]")
- OmicsDI_search_datasets: Cross-repository dataset search
- SCXA_list_experiments / SCXA_search_gene: EBI Single Cell Expression Atlas — 380+ curated scRNA-seq datasets, searchable by species/keyword or by which experiments show a gene expressed.
SCXA_search_gene can be slow (10-40s+ per call observed live) — don't assume it failed just because it takes longer than other lookups.
- SCXA_get_cluster_marker_genes: Pre-computed per-cluster/per-cell-type marker genes for a specific SCXA experiment.
marker_type only accepts "clusters" or "cell_types" (not "cell_type"), and marker_type="cell_types" additionally requires organism_part (e.g. "pancreas", "lung") — verified live, both are hard parameter-validation errors, not soft warnings.
- SCP_search_studies / SCP_list_studies / SCP_get_study: Broad Institute Single Cell Portal — 1000+ public studies, 80M+ cells.
SCP_list_studies sorts by cell count descending and can filter by min_cells.
- UCSCCellBrowser_search_datasets / UCSCCellBrowser_get_dataset / UCSCCellBrowser_list_facets: 300+ curated single-cell datasets filterable by organism/body part/disease. Call
UCSCCellBrowser_list_facets first to get valid filter values (facets are "organisms", "body_parts", "diseases") before filtering search_datasets.
Cell Type Markers
- CellMarker_search_by_cell_type: Tissue-specific cell markers (use
CellMarker_list_cell_types first — exact names required, e.g., "Regulatory T(Treg) cell" not "Regulatory T cell")
- CellMarker_search_cancer_markers: Cancer-context markers with experimental evidence
- CellMarker_search_by_gene: Reverse lookup — which cell types express a gene?
- HPA_search_genes_by_query: Cell-type marker gene search
- PanglaoDB_list_cell_types / PanglaoDB_markers_for_cell_type / PanglaoDB_cell_types_for_gene: PanglaoDB curated marker genes, each with a
canonical_marker flag plus per-species sensitivity_*/specificity_* scores. Call PanglaoDB_list_cell_types (optionally filtered by organ) to get exact valid cell_type strings first. Results from PanglaoDB_cell_types_for_gene are NOT ranked by specificity — verified live, querying CD3E (a canonical T-cell marker) returned "Macrophages" as the first hit with canonical_marker: true and only a low specificity_human: 0.1; always check canonical_marker and the specificity_*/sensitivity_* fields per row rather than trusting result order, and cross-check against CellMarker/HPA above for annotation-critical calls.
- CellTypist_search_models / CellTypist_get_model: catalog of 60+ pre-trained CellTypist classifiers for automated cell-type annotation (as opposed to marker-gene lookup above) — filter by
keyword (tissue/condition, e.g. "lung", "immune", "fetal") and/or min_celltypes, or fetch one model's full metadata by exact filename (e.g. "Immune_All_Low.pkl", the CellTypist default, 98 cell types). Each entry gives n_celltypes, source_doi, and a download_url for the serialized model — this is catalog metadata only, not annotation output. To actually run a chosen model against an expression matrix, use the remote tool run_celltypist_annotate. Reach for this when the question is "which pre-trained classifier exists for tissue X" rather than "what genes mark cell type Y" (PanglaoDB/CellMarker above).
Gene Annotation
- MyGene_query_genes / MyGene_batch_query: Gene ID conversion
- ensembl_lookup_gene: Ensembl gene details
- UniProt_get_function_by_accession: Protein function
Cell-Cell Communication
- OmniPath_get_ligand_receptor_interactions: L-R pairs (CellPhoneDB, CellChatDB)
- OmniPath_get_signaling_interactions: Downstream signaling
- OmniPath_get_complexes: Multi-subunit receptors
Enrichment (Post-DE)
- PANTHER_enrichment: GO enrichment (BP, MF, CC)
- STRING_functional_enrichment: Network-based enrichment
- ReactomeAnalysis_pathway_enrichment: Reactome pathways
Clinical Context (for tumor immunology)
- DGIdb_get_drug_gene_interactions: Drug interactions for immune checkpoint targets (genes=["CD274"] for PD-L1)
- civic_search_evidence_items: Clinical evidence for mutations/biomarkers
- TIMER2_immune_estimation: TCGA immune infiltration correlation
- search_clinical_trials: Clinical trial matching
- GTEx_get_expression_summary: Normal tissue baseline expression
- PubMed_search_articles: Literature context
Scanpy vs Seurat Equivalents
| Operation |
Seurat (R) |
Scanpy (Python) |
| Load data |
Read10X() |
sc.read_10x_mtx() |
| Normalize |
NormalizeData() |
sc.pp.normalize_total() + sc.pp.log1p() |
| Find HVGs |
FindVariableFeatures() |
sc.pp.highly_variable_genes() |
| PCA |
RunPCA() |
sc.tl.pca() |
| Cluster |
FindClusters() |
sc.tl.leiden() |
| UMAP |
RunUMAP() |
sc.tl.umap() |
| Find markers |
FindMarkers() |
sc.tl.rank_genes_groups() |
| Batch correction |
RunHarmony() |
harmonypy.run_harmony() |
Reasoning Framework for Result Interpretation
Evidence Grading
| Grade |
Criteria |
Example |
| High confidence |
Marker padj < 0.01, log2FC > 1, expressed in > 25% of cluster cells |
CD3D as T-cell marker with padj = 1e-50, log2FC = 3.2, pct = 0.85 |
| Moderate confidence |
padj < 0.05, log2FC > 0.5, or expressed in 10-25% of cluster |
FOXP3 in Treg cluster with padj = 0.001, pct = 0.18 |
| Low confidence |
padj < 0.05 but log2FC < 0.5 or low pct_diff between clusters |
Ubiquitously expressed gene with marginal enrichment |
| Unreliable |
Fewer than 20 cells in cluster, or QC metrics suggest doublets |
Cluster with mean nGenes > 6000 and high doublet score |
Interpretation Guidance
- QC metric thresholds: Standard filters are nGenes > 200 (remove empty droplets), nGenes < 5000-6000 (remove doublets), pct_counts_mt < 20% (remove dying cells). These thresholds are tissue-dependent: immune cells tolerate stricter nGene filters; neurons may have higher mitochondrial content naturally. Always visualize distributions before setting cutoffs.
- Cluster resolution guidance: Leiden resolution 0.3-0.5 yields broad cell types (T cells, B cells, myeloid). Resolution 0.8-1.2 resolves subtypes (CD4 naive, CD4 memory, Treg). Resolution > 2.0 risks over-clustering (splitting biologically homogeneous populations). Validate by checking that each cluster has distinct marker genes.
- Marker gene confidence levels: A strong marker is highly specific (high pct_diff between cluster and rest) and highly expressed (high log2FC). Genes expressed in many clusters with small fold changes are poor markers. Cross-reference with known markers from CellMarker or HPA databases.
- Pseudo-bulk vs single-cell DE: For comparing conditions (treatment vs control), pseudo-bulk DE (aggregate by sample, then DESeq2) is more statistically valid than single-cell DE, which inflates significance due to non-independence of cells from the same sample.
- Batch effects: If samples cluster by batch rather than biology on UMAP, apply Harmony or other correction before biological interpretation.
Synthesis Questions
- Do the identified clusters correspond to known cell types based on canonical markers, or do some clusters lack clear biological identity (potentially doublets or low-quality cells)?
- At the chosen clustering resolution, are there clusters that merge when resolution is lowered, suggesting they may be a single cell type split by technical noise?
- For differential expression between conditions, are the results consistent between single-cell and pseudo-bulk approaches, and do the top DE genes have known biological relevance?
- Do QC-flagged cells (high mito, extreme gene counts) concentrate in specific clusters, and does removing them change the clustering structure?
- If batch correction was applied, do post-correction clusters still maintain expected cell-type-specific marker expression?
Troubleshooting
| Issue |
Solution |
ModuleNotFoundError: leidenalg |
pip install leidenalg |
| Sparse matrix errors |
.toarray(): X = adata.X.toarray() if issparse(adata.X) else adata.X |
| Wrong matrix orientation |
More genes than samples? Transpose |
| NaN in correlation |
Filter: valid = ~np.isnan(x) & ~np.isnan(y) |
| Too few cells for DE |
Need >= 3 cells per condition per cell type |
| Memory error |
Use sc.pp.highly_variable_genes() to reduce features |
Reference Documentation
Detailed Analysis Patterns: analysis_patterns.md (per-cell-type DE, correlation, PCA, ANOVA, cell communication)
Core Workflows:
- references/scrna_qc.md - scRNA-seq QC gating (mito%, doublets, ambient RNA, empty droplets, MAD thresholds)
- references/scanpy_workflow.md - Complete scanpy pipeline
- references/seurat_workflow.md - Seurat to Scanpy translation
- references/clustering_guide.md - Clustering methods
- references/marker_identification.md - Marker genes, annotation
- references/trajectory_analysis.md - Pseudotime
- references/cell_communication.md - OmniPath/CellPhoneDB workflow
- references/troubleshooting.md - Detailed error solutions
Analysis Conventions
DESeq2 library choice: match the authoritative script
If the data folder contains an authoritative script (run_*.py, analysis.R), use whichever DESeq2 library it uses (pydeseq2 or R DESeq2). The two libraries give slightly different DEG counts (~2-10% at the same thresholds), so matching matters. If no script exists, prefer R DESeq2 via the run_deseq2_analysis tool or Rscript:
tu run run_deseq2_analysis '{"operation":"deseq2","counts_file":"pseudo_bulk_counts.csv","metadata_file":"sample_meta.csv","design":"~ sex","contrast":"sex, M, F","lfc_shrinkage":true}'
1---23name: tooluniverse-single-cell-24description: "Single-cell RNA-seq analysis with scanpy/anndata — h5ad data loading, scRNA-seq quality control and QC gating (n_genes_by_counts, total_counts, mitochondrial percent / pct_counts_mt, pct_counts_ribo, doublet detection with Scrublet/scDblFinder, ambient RNA / SoupX awareness, empty-droplet filtering, MAD-based thresholds), normalization, dimensionality reduction (PCA, UMAP, t-SNE), clustering (Leiden, Louvain), marker gene identification, cell-type annotation, pseudotime/trajectory analysis. Use for any scRNA-seq workflow, including deciding which cells to filter, flag, or investigate before downstream analysis."5metadata:6 source: https://skillmd.com/skills/mims-harvard/tooluniverse-single-cell-27---89# Single-Cell Genomics and Expression Matrix Analysis1011## RULE ZERO — Check for pre-computed results FIRST1213Before following any instruction below, scan the data folder for:14- `*_executed.ipynb` → read with `tu run read_executed_notebook '{"data_folder":"<path>","search":"<keyword>"}'` and cite its cell outputs as the authoritative answer15- Pre-computed result files (CSV/TSV with names like `*results*`, `*deseq*`, `*enrich*`, `*stats*`, `*_simplified.csv`) → read directly and report the requested value16- Canonical analysis scripts (`analysis.R`, `run_*.py`, `find_*.R`, `*.Rmd`) → execute as-is and read the output1718Only follow this skill's re-analysis recipe below if **none** of the above exist. Re-running from raw data produces different numbers than the published answer and is much slower (often 5-10× turn count).1920---2122Comprehensive single-cell RNA-seq analysis and expression matrix processing using scanpy, anndata, scipy, and ToolUniverse.2324---2526## LOOK UP, DON'T GUESS27When uncertain about any scientific fact, SEARCH databases first (PubMed, UniProt, ChEMBL, ClinVar, etc.) rather than reasoning from memory. A database-verified answer is always more reliable than a guess.2829---3031## When to Use This Skill3233Apply when users:34- Have scRNA-seq data (h5ad, 10X, CSV count matrices) and want analysis35- Need scRNA-seq quality control / QC gating: deciding cell filters by36 mito % (pct_counts_mt), gene/UMI counts, doublets, ambient RNA, empty droplets37- Ask about cell type identification, clustering, or annotation38- Need differential expression analysis by cell type or condition39- Want gene-expression correlation analysis (e.g., gene length vs expression by cell type)40- Ask about PCA, UMAP, t-SNE for expression data41- Need Leiden/Louvain clustering on expression matrices42- Want statistical comparisons between cell types (t-test, ANOVA, fold change)43- Ask about marker genes, batch correction, trajectory, or cell-cell communication4445**NOT for** (use other skills instead):46- Bulk RNA-seq DESeq2 only -> `tooluniverse-rnaseq-deseq2`47- Gene enrichment only -> `tooluniverse-gene-enrichment`48- VCF/variant analysis -> `tooluniverse-variant-analysis`49- Deep-learning integration/batch correction (scVI, scANVI, totalVI, PeakVI, MultiVI, DestVI, veloVI) -> `tooluniverse-scvi-tools`5051---5253## Core Principles54551. **Data-first** - Load, inspect, validate before analysis562. **AnnData-centric** - All data flows through anndata objects573. **Cell type awareness** - Per-cell-type subsetting when needed584. **Statistical rigor** - Normalization, multiple testing correction, effect sizes595. **Question-driven** - Parse what the user is actually asking6061---6263## Required Packages6465```python66import scanpy as sc, anndata as ad, pandas as pd, numpy as np67from scipy import stats68from scipy.cluster.hierarchy import linkage, fcluster69from sklearn.decomposition import PCA70from statsmodels.stats.multitest import multipletests71import gseapy as gp # enrichment72import harmonypy # batch correction (optional)73```7475Install: `pip install scanpy anndata leidenalg umap-learn harmonypy gseapy pandas numpy scipy scikit-learn statsmodels`7677---7879## Workflow Decision Tree8081```82START: User question about scRNA-seq data83|84+-- FULL PIPELINE (raw counts -> annotated clusters)85| Workflow: QC -> Normalize -> HVG -> PCA -> Cluster -> Annotate -> DE86| See: references/scanpy_workflow.md87|88+-- DIFFERENTIAL EXPRESSION (per-cell-type comparison)89| Most common pattern: per-cell-type DE90| See: analysis_patterns.md "Pattern 1"91|92+-- CORRELATION ANALYSIS (gene property vs expression)93| Pattern: Gene length vs expression correlation94| See: analysis_patterns.md "Pattern 2"95|96+-- CLUSTERING & PCA (expression matrix analysis)97| See: references/clustering_guide.md98|99+-- CELL COMMUNICATION (ligand-receptor interactions)100| See: references/cell_communication.md101|102+-- TRAJECTORY ANALYSIS (pseudotime)103 See: references/trajectory_analysis.md104```105106**Data format handling**:107- h5ad -> `sc.read_h5ad()`108- 10X -> `sc.read_10x_mtx()` or `sc.read_10x_h5()`109- CSV/TSV -> `pd.read_csv()` -> Convert to AnnData (check orientation!)110111---112113## Data Loading114115AnnData expects: **cells/samples as rows (obs), genes as columns (var)**116117```python118adata = sc.read_h5ad("data.h5ad") # h5ad already oriented119120# CSV/TSV: check orientation121df = pd.read_csv("counts.csv", index_col=0)122if df.shape[0] > df.shape[1] * 5: # genes > samples by 5x => transpose123 df = df.T124adata = ad.AnnData(df)125126# Load metadata127meta = pd.read_csv("metadata.csv", index_col=0)128common = adata.obs_names.intersection(meta.index)129adata = adata[common].copy()130for col in meta.columns:131 adata.obs[col] = meta.loc[common, col]132```133134---135136## Quality Control and QC Gating (do this BEFORE downstream analysis)137138QC gating decides **which cells and genes are real** before normalization,139clustering, or DE. Skipping or rushing it propagates silently: doublets become140fake "intermediate" states, ambient RNA smears markers across clusters, and141empty droplets inflate cell counts. Never report a filtered cell count without142the gates applied, and never report cutoffs you did not actually run.143144**HONEST EXECUTION**: QC runs scanpy/AnnData via Bash/Python. If scanpy is not145installed, do NOT fabricate metrics — print the install plan and stop:146`python scripts/scrna_qc.py --install-plan` (exits 0, lists what is missing,147suggests `pip install scanpy anndata scrublet`).148149### Per-cell QC metrics (what to compute)150151```python152vn = adata.var_names.str.upper()153adata.var['mt'] = vn.str.startswith('MT-') # mitochondrial154adata.var['ribo'] = vn.str.startswith(('RPS', 'RPL')) # ribosomal protein155adata.var['hb'] = vn.str.contains(r'^HB[^P]', regex=True) # hemoglobin (RBC)156sc.pp.calculate_qc_metrics(157 adata, qc_vars=['mt', 'ribo', 'hb'],158 percent_top=None, # REQUIRED for small gene panels (<500) — else IndexError159 log1p=True, inplace=True)160```161162Key metrics in `adata.obs`: `n_genes_by_counts`, `total_counts`,163`pct_counts_mt`, `pct_counts_ribo`, `pct_counts_hb`.164165### WHY each metric flags a problem (reason from biology, not magic numbers)166167- **High `pct_counts_mt` -> dying / stressed cell.** A ruptured membrane lets168 cytoplasmic mRNA leak out while mito transcripts stay trapped, enriching the169 captured RNA for mito. Cutoff is tissue-dependent (cardiomyocytes/hepatocytes170 are mito-rich at baseline — a blanket 10% would discard healthy cells).171- **Low `n_genes_by_counts` / `total_counts` -> empty droplet or debris**172 (only ambient RNA captured; few genes, low depth).173- **Very high counts/genes -> doublet** (two transcriptomes ~double depth and174 diversity) — but high count alone is weak; use a doublet caller (below).175- **High `pct_counts_hb` -> RBC/blood contamination** in solid tissue.176177### Choosing thresholds — distribution-aware (MAD), not hardcoded178179Hardcoded cutoffs (`mt<5%`, `n_genes<2500`) are a starting point only; they180break on mito-rich tissues and on shallow vs deep libraries. Prefer a181**MAD-based** rule (robust to the very outliers you are removing): flag cells182> `nmads` median-absolute-deviations from the median. Use `nmads=5` on183log1p counts/genes (both tails), `nmads=3` upper-only on `pct_counts_mt`, and184pair mito with a biological ceiling so a uniformly degraded sample doesn't pass.185**Always visualize distributions first** (violin + `total_counts` vs186`pct_counts_mt` scatter — dying cells sit in the low-count/high-mito corner).187188Run the helper (computes metrics + MAD gating, reports per-step removals):189190```bash191python scripts/scrna_qc.py data.h5ad --doublets # or --install-plan first192```193194### Doublets, ambient RNA, empty droplets (per-cell metrics miss these)195196- **Doublets**: Scrublet (`sc.pp.scrublet`, scanpy >=1.10) or scDblFinder (R).197 Run **per sample** before merging; flag-cluster-drop (doublets form bridge198 clusters). `expected_doublet_rate` ~0.8%/1,000 cells recovered (10x).199- **Ambient RNA**: cell-free "soup" mRNA in every droplet — a count-correction200 step (SoupX / DecontX, R), NOT a cell filter. Per-cell QC cannot detect it;201 suspect it when markers look implausibly ubiquitous.202- **Empty droplets**: upstream of per-cell QC. CellRanger's filtered matrix203 already applies an EmptyDrops-style call; with only the raw matrix, run204 EmptyDrops (DropletUtils, R) or a barcode-rank knee before per-cell QC.205206### QC Interpretation Table (metric -> concern -> action)207208| QC metric | Direction | Typical concern | Suggested action |209|-----------|-----------|-----------------|------------------|210| `pct_counts_mt` | high | Dying / stressed cell (membrane rupture) | **Filter** (MAD upper + tissue-aware ceiling; raise ceiling for mito-rich tissue) |211| `n_genes_by_counts` | very low | Empty droplet / debris | **Filter** (`min_genes` ~200 + low-tail MAD) |212| `total_counts` | very low | Shallow / failed capture | **Filter** (low-tail MAD; check barcode-rank knee) |213| `n_genes_by_counts` / `total_counts` | very high | Doublet (two cells in one droplet) | **Flag**, run Scrublet/scDblFinder, then drop — don't hard-cap on counts alone |214| `predicted_doublet` (Scrublet) | True | Multiplet | **Filter** per sample; cluster-then-drop if unsure |215| `pct_counts_hb` | high | RBC / blood contamination | **Filter** in non-blood tissue; **investigate** in blood |216| `pct_counts_ribo` | very high/low | Low-complexity / stressed, or cell-type signal | **Flag / investigate** (ribo is cell-type-specific; rarely a hard filter) |217| markers ubiquitous across clusters | — | Ambient RNA contamination | **Investigate** — run SoupX/DecontX, do not silently proceed |218| many cells, low median counts | — | Empty droplets not removed | **Investigate** — apply EmptyDrops / knee, re-filter |219220Full reasoning, MAD code, Scrublet/SoupX/EmptyDrops recipes, and order of221operations: **references/scrna_qc.md**. Inline pipeline QC:222**references/scanpy_workflow.md** Phase 2.223224---225226## Complete Pipeline (Quick Reference)227228```python229import scanpy as sc230231adata = sc.read_10x_h5("filtered_feature_bc_matrix.h5")232233# QC234adata.var['mt'] = adata.var_names.str.startswith('MT-')235sc.pp.calculate_qc_metrics(adata, qc_vars=['mt'], inplace=True)236adata = adata[adata.obs['pct_counts_mt'] < 20].copy()237sc.pp.filter_cells(adata, min_genes=200)238sc.pp.filter_genes(adata, min_cells=3)239240# Normalize + HVG + PCA241sc.pp.normalize_total(adata, target_sum=1e4)242sc.pp.log1p(adata)243adata.raw = adata.copy()244sc.pp.highly_variable_genes(adata, n_top_genes=2000)245sc.tl.pca(adata, n_comps=50)246247# Cluster + UMAP248sc.pp.neighbors(adata, n_pcs=30)249sc.tl.leiden(adata, resolution=0.5)250sc.tl.umap(adata)251252# Find markers + Annotate + Per-cell-type DE253sc.tl.rank_genes_groups(adata, groupby='leiden', method='wilcoxon')254```255256---257258## Differential Expression Decision Tree259260```261Single-Cell DE (many cells per condition):262 Use: sc.tl.rank_genes_groups(), methods: wilcoxon, t-test, logreg263 Best for: Per-cell-type DE, marker gene finding264265Pseudo-Bulk DE (aggregate counts by sample):266 Use: R DESeq2 via `tu run run_deseq2_analysis` or Rscript (NOT pydeseq2 — gives different DEG counts)267 Best for: Sample-level comparisons with replicates268269Statistical Tests Only:270 Use: scipy.stats (ttest_ind, f_oneway, pearsonr)271 Best for: Correlation, ANOVA, t-tests on summaries272```273274---275276## Statistical Tests (Quick Reference)277278```python279from scipy import stats280from statsmodels.stats.multitest import multipletests281282# Pearson/Spearman correlation283r, p = stats.pearsonr(gene_lengths, mean_expression)284285# Welch's t-test286t_stat, p_val = stats.ttest_ind(group1, group2, equal_var=False)287288# ANOVA289f_stat, p_val = stats.f_oneway(group1, group2, group3)290291# Multiple testing correction (BH)292reject, pvals_adj, _, _ = multipletests(pvals, method='fdr_bh')293```294295---296297## Batch Correction (Harmony)298299```python300import harmonypy301sc.tl.pca(adata, n_comps=50)302ho = harmonypy.run_harmony(adata.obsm['X_pca'][:, :30], adata.obs, 'batch', random_state=0)303adata.obsm['X_pca_harmony'] = ho.Z_corr.T304sc.pp.neighbors(adata, use_rep='X_pca_harmony')305sc.tl.leiden(adata, resolution=0.5)306sc.tl.umap(adata)307```308309---310311## ToolUniverse Integration312313### Data Discovery (before analysis)314- **CxGDisc_search_datasets**: Search CELLxGENE Discover for scRNA-seq datasets by disease, tissue, organism. Use broad disease terms (e.g., "breast cancer" not "triple-negative").315- **GEO_search_rnaseq_datasets** / **geo_search_datasets**: Search GEO for scRNA-seq studies316- **NCBI_SRA_search_runs**: Search SRA for sequencing runs (query="single cell RNA-seq [disease]")317- **OmicsDI_search_datasets**: Cross-repository dataset search318- **SCXA_list_experiments** / **SCXA_search_gene**: EBI Single Cell Expression Atlas — 380+ curated scRNA-seq datasets, searchable by species/keyword or by which experiments show a gene expressed. `SCXA_search_gene` can be slow (10-40s+ per call observed live) — don't assume it failed just because it takes longer than other lookups.319- **SCXA_get_cluster_marker_genes**: Pre-computed per-cluster/per-cell-type marker genes for a specific SCXA experiment. `marker_type` only accepts `"clusters"` or `"cell_types"` (not `"cell_type"`), and `marker_type="cell_types"` additionally *requires* `organism_part` (e.g. "pancreas", "lung") — verified live, both are hard parameter-validation errors, not soft warnings.320- **SCP_search_studies** / **SCP_list_studies** / **SCP_get_study**: Broad Institute Single Cell Portal — 1000+ public studies, 80M+ cells. `SCP_list_studies` sorts by cell count descending and can filter by `min_cells`.321- **UCSCCellBrowser_search_datasets** / **UCSCCellBrowser_get_dataset** / **UCSCCellBrowser_list_facets**: 300+ curated single-cell datasets filterable by organism/body part/disease. Call `UCSCCellBrowser_list_facets` first to get valid filter values (facets are `"organisms"`, `"body_parts"`, `"diseases"`) before filtering `search_datasets`.322323### Cell Type Markers324- **CellMarker_search_by_cell_type**: Tissue-specific cell markers (use `CellMarker_list_cell_types` first — exact names required, e.g., "Regulatory T(Treg) cell" not "Regulatory T cell")325- **CellMarker_search_cancer_markers**: Cancer-context markers with experimental evidence326- **CellMarker_search_by_gene**: Reverse lookup — which cell types express a gene?327- **HPA_search_genes_by_query**: Cell-type marker gene search328- **PanglaoDB_list_cell_types** / **PanglaoDB_markers_for_cell_type** / **PanglaoDB_cell_types_for_gene**: PanglaoDB curated marker genes, each with a `canonical_marker` flag plus per-species `sensitivity_*`/`specificity_*` scores. Call `PanglaoDB_list_cell_types` (optionally filtered by `organ`) to get exact valid `cell_type` strings first. **Results from `PanglaoDB_cell_types_for_gene` are NOT ranked by specificity** — verified live, querying `CD3E` (a canonical T-cell marker) returned "Macrophages" as the *first* hit with `canonical_marker: true` and only a low `specificity_human: 0.1`; always check `canonical_marker` and the `specificity_*`/`sensitivity_*` fields per row rather than trusting result order, and cross-check against CellMarker/HPA above for annotation-critical calls.329- **CellTypist_search_models** / **CellTypist_get_model**: catalog of 60+ pre-trained CellTypist classifiers for *automated* cell-type annotation (as opposed to marker-gene lookup above) — filter by `keyword` (tissue/condition, e.g. "lung", "immune", "fetal") and/or `min_celltypes`, or fetch one model's full metadata by exact `filename` (e.g. `"Immune_All_Low.pkl"`, the CellTypist default, 98 cell types). Each entry gives `n_celltypes`, `source_doi`, and a `download_url` for the serialized model — this is catalog metadata only, not annotation output. To actually run a chosen model against an expression matrix, use the remote tool `run_celltypist_annotate`. Reach for this when the question is "which pre-trained classifier exists for tissue X" rather than "what genes mark cell type Y" (PanglaoDB/CellMarker above).330331### Gene Annotation332- **MyGene_query_genes** / **MyGene_batch_query**: Gene ID conversion333- **ensembl_lookup_gene**: Ensembl gene details334- **UniProt_get_function_by_accession**: Protein function335336### Cell-Cell Communication337- **OmniPath_get_ligand_receptor_interactions**: L-R pairs (CellPhoneDB, CellChatDB)338- **OmniPath_get_signaling_interactions**: Downstream signaling339- **OmniPath_get_complexes**: Multi-subunit receptors340341### Enrichment (Post-DE)342- **PANTHER_enrichment**: GO enrichment (BP, MF, CC)343- **STRING_functional_enrichment**: Network-based enrichment344- **ReactomeAnalysis_pathway_enrichment**: Reactome pathways345346### Clinical Context (for tumor immunology)347- **DGIdb_get_drug_gene_interactions**: Drug interactions for immune checkpoint targets (genes=["CD274"] for PD-L1)348- **civic_search_evidence_items**: Clinical evidence for mutations/biomarkers349- **TIMER2_immune_estimation**: TCGA immune infiltration correlation350- **search_clinical_trials**: Clinical trial matching351- **GTEx_get_expression_summary**: Normal tissue baseline expression352- **PubMed_search_articles**: Literature context353354---355356## Scanpy vs Seurat Equivalents357358| Operation | Seurat (R) | Scanpy (Python) |359|-----------|------------|-----------------|360| Load data | `Read10X()` | `sc.read_10x_mtx()` |361| Normalize | `NormalizeData()` | `sc.pp.normalize_total() + sc.pp.log1p()` |362| Find HVGs | `FindVariableFeatures()` | `sc.pp.highly_variable_genes()` |363| PCA | `RunPCA()` | `sc.tl.pca()` |364| Cluster | `FindClusters()` | `sc.tl.leiden()` |365| UMAP | `RunUMAP()` | `sc.tl.umap()` |366| Find markers | `FindMarkers()` | `sc.tl.rank_genes_groups()` |367| Batch correction | `RunHarmony()` | `harmonypy.run_harmony()` |368369---370371## Reasoning Framework for Result Interpretation372373### Evidence Grading374375| Grade | Criteria | Example |376|-------|----------|---------|377| **High confidence** | Marker padj < 0.01, log2FC > 1, expressed in > 25% of cluster cells | CD3D as T-cell marker with padj = 1e-50, log2FC = 3.2, pct = 0.85 |378| **Moderate confidence** | padj < 0.05, log2FC > 0.5, or expressed in 10-25% of cluster | FOXP3 in Treg cluster with padj = 0.001, pct = 0.18 |379| **Low confidence** | padj < 0.05 but log2FC < 0.5 or low pct_diff between clusters | Ubiquitously expressed gene with marginal enrichment |380| **Unreliable** | Fewer than 20 cells in cluster, or QC metrics suggest doublets | Cluster with mean nGenes > 6000 and high doublet score |381382### Interpretation Guidance383384- **QC metric thresholds**: Standard filters are nGenes > 200 (remove empty droplets), nGenes < 5000-6000 (remove doublets), pct_counts_mt < 20% (remove dying cells). These thresholds are tissue-dependent: immune cells tolerate stricter nGene filters; neurons may have higher mitochondrial content naturally. Always visualize distributions before setting cutoffs.385- **Cluster resolution guidance**: Leiden resolution 0.3-0.5 yields broad cell types (T cells, B cells, myeloid). Resolution 0.8-1.2 resolves subtypes (CD4 naive, CD4 memory, Treg). Resolution > 2.0 risks over-clustering (splitting biologically homogeneous populations). Validate by checking that each cluster has distinct marker genes.386- **Marker gene confidence levels**: A strong marker is highly specific (high pct_diff between cluster and rest) and highly expressed (high log2FC). Genes expressed in many clusters with small fold changes are poor markers. Cross-reference with known markers from CellMarker or HPA databases.387- **Pseudo-bulk vs single-cell DE**: For comparing conditions (treatment vs control), pseudo-bulk DE (aggregate by sample, then DESeq2) is more statistically valid than single-cell DE, which inflates significance due to non-independence of cells from the same sample.388- **Batch effects**: If samples cluster by batch rather than biology on UMAP, apply Harmony or other correction before biological interpretation.389390### Synthesis Questions3913921. Do the identified clusters correspond to known cell types based on canonical markers, or do some clusters lack clear biological identity (potentially doublets or low-quality cells)?3932. At the chosen clustering resolution, are there clusters that merge when resolution is lowered, suggesting they may be a single cell type split by technical noise?3943. For differential expression between conditions, are the results consistent between single-cell and pseudo-bulk approaches, and do the top DE genes have known biological relevance?3954. Do QC-flagged cells (high mito, extreme gene counts) concentrate in specific clusters, and does removing them change the clustering structure?3965. If batch correction was applied, do post-correction clusters still maintain expected cell-type-specific marker expression?397398---399400## Troubleshooting401402| Issue | Solution |403|-------|----------|404| `ModuleNotFoundError: leidenalg` | `pip install leidenalg` |405| Sparse matrix errors | `.toarray()`: `X = adata.X.toarray() if issparse(adata.X) else adata.X` |406| Wrong matrix orientation | More genes than samples? Transpose |407| NaN in correlation | Filter: `valid = ~np.isnan(x) & ~np.isnan(y)` |408| Too few cells for DE | Need >= 3 cells per condition per cell type |409| Memory error | Use `sc.pp.highly_variable_genes()` to reduce features |410411---412413## Reference Documentation414415**Detailed Analysis Patterns**: analysis_patterns.md (per-cell-type DE, correlation, PCA, ANOVA, cell communication)416417**Core Workflows**:418- references/scrna_qc.md - scRNA-seq QC gating (mito%, doublets, ambient RNA, empty droplets, MAD thresholds)419- references/scanpy_workflow.md - Complete scanpy pipeline420- references/seurat_workflow.md - Seurat to Scanpy translation421- references/clustering_guide.md - Clustering methods422- references/marker_identification.md - Marker genes, annotation423- references/trajectory_analysis.md - Pseudotime424- references/cell_communication.md - OmniPath/CellPhoneDB workflow425- references/troubleshooting.md - Detailed error solutions426427---428429## Analysis Conventions430431### DESeq2 library choice: match the authoritative script432If the data folder contains an authoritative script (`run_*.py`, `analysis.R`), use whichever DESeq2 library it uses (pydeseq2 or R DESeq2). The two libraries give slightly different DEG counts (~2-10% at the same thresholds), so matching matters. If no script exists, prefer R DESeq2 via the `run_deseq2_analysis` tool or `Rscript`:433434```bash435tu run run_deseq2_analysis '{"operation":"deseq2","counts_file":"pseudo_bulk_counts.csv","metadata_file":"sample_meta.csv","design":"~ sex","contrast":"sex, M, F","lfc_shrinkage":true}'436```