Source: https://github.com/aipoch/medical-research-skills
When to Use
Use AnnData when you need to:
- Load, inspect, or export annotated single-cell datasets stored as
.h5ad (or zarr) for downstream tools.
- Keep a matrix (cells × features) tightly coupled with observation/feature metadata (e.g., cell types, batches, gene annotations).
- Work efficiently with large, sparse count matrices (e.g., scRNA-seq) and avoid loading everything into memory (backed mode).
- Combine multiple experiments/batches/modalities into a unified object while tracking provenance.
- Subset/filter/transform data while preserving alignment between the matrix and metadata.
Key Features
- Unified container:
X (data matrix) plus aligned annotations: obs, var, uns, and multi-dimensional slots (obsm, varm, obsp, varp), plus layers and optional raw.
- Interoperable I/O: Native
.h5ad and zarr, plus common genomics formats (e.g., 10x, loom, mtx, csv).
- Scalable workflows: Sparse matrices and backed mode (
backed="r") for large datasets.
- Safe subsetting: Slicing preserves alignment across matrix and annotations; supports views vs copies.
- Concatenation utilities:
ad.concat(...) with join/merge strategies and batch labeling; experimental lazy collections.
Reference notes: the original material mentions additional guides under references/ (e.g., references/data_structure.md, references/io_operations.md, references/concatenation.md, references/manipulation.md, references/best_practices.md) for deeper explanations of each topic.
Dependencies
anndata (latest compatible with your environment; install via pip/uv)
numpy
pandas
scipy (recommended for sparse matrices)
- Optional ecosystem tools (only if needed):
scanpy
muon
torch (for deep learning) and anndata experimental loader utilities
Example Usage
A complete runnable example that creates an AnnData object, writes/reads .h5ad, subsets, concatenates batches, and demonstrates backed mode.
import numpy as np
import pandas as pd
import anndata as ad
from scipy.sparse import csr_matrix
# ----------------------------
# 1) Create an AnnData object
# ----------------------------
rng = np.random.default_rng(0)
n_cells, n_genes = 100, 500
X = rng.poisson(1.0, size=(n_cells, n_genes)).astype(np.float32)
obs = pd.DataFrame(
{
"cell_type": (["T cell", "B cell"] * (n_cells // 2)),
"sample": (["A", "B"] * (n_cells // 2)),
"quality_score": rng.random(n_cells),
},
index=[f"cell_{i}" for i in range(n_cells)],
)
var = pd.DataFrame(
{"gene_name": [f"Gene_{j}" for j in range(n_genes)]},
index=[f"ENSG{j:05d}" for j in range(n_genes)],
)
adata = ad.AnnData(X=X, obs=obs, var=var)
# Use sparse storage for typical count-like matrices
adata.X = csr_matrix(adata.X)
# Convert string columns to categoricals to reduce memory and speed up ops
adata.strings_to_categoricals()
print(f"Created: {adata.n_obs} obs × {adata.n_vars} vars")
# ----------------------------
# 2) Write and read .h5ad
# ----------------------------
adata.write_h5ad("example.h5ad", compression="gzip")
adata2 = ad.read_h5ad("example.h5ad")
print(f"Reloaded: {adata2.n_obs} obs × {adata2.n_vars} vars")
# ----------------------------
# 3) Subset (keeps alignment)
# ----------------------------
t_cells = adata2[adata2.obs["cell_type"] == "T cell", :]
high_quality = adata2[adata2.obs["quality_score"] > 0.8, :]
print(f"T cells: {t_cells.n_obs}")
print(f"High quality: {high_quality.n_obs}")
# ----------------------------
# 4) Concatenate batches
# ----------------------------
adata_a = adata2[adata2.obs["sample"] == "A", :].copy()
adata_b = adata2[adata2.obs["sample"] == "B", :].copy()
combined = ad.concat(
[adata_a, adata_b],
axis=0, # concatenate observations (cells)
join="inner", # keep shared variables
label="batch", # add a column in .obs
keys=["A", "B"], # batch labels
)
print(combined.obs["batch"].value_counts().to_dict())
# ----------------------------
# 5) Backed mode for large files
# ----------------------------
adata_backed = ad.read_h5ad("example.h5ad", backed="r")
# Slicing in backed mode is metadata-friendly; load to memory when needed:
subset_mem = adata_backed[:10, :50].to_memory()
print(f"Backed subset loaded: {subset_mem.shape}")
Implementation Details
Data model (core slots)
X: primary data matrix (dense numpy.ndarray or sparse scipy.sparse), shape (n_obs, n_vars).
obs: per-observation metadata (pandas.DataFrame), indexed by obs_names (e.g., cell IDs).
var: per-variable metadata (pandas.DataFrame), indexed by var_names (e.g., gene IDs).
layers: named alternative matrices aligned to X (e.g., "counts", "log1p").
obsm / varm: multi-dimensional embeddings aligned to obs/var (e.g., PCA, UMAP coordinates).
obsp / varp: pairwise graphs/matrices (e.g., kNN graph in obsp["connectivities"]).
uns: unstructured metadata (dict-like), often used for parameters and plotting configs.
raw (optional): snapshot of unfiltered/untransformed data for reproducibility.
Views vs copies
- Slicing like
adata_subset = adata[mask, :] typically returns a view (lightweight reference).
- Use
.copy() when you need an independent object (e.g., before in-place modifications).
Backed mode (large datasets)
ad.read_h5ad(path, backed="r") keeps the matrix on disk and loads data lazily.
- Convert a slice to memory with
.to_memory() when you need in-memory computation.
- Backed mode is best for filtering by metadata, chunked processing, and avoiding OOM.
Concatenation behavior
ad.concat([...], axis=0) stacks observations; axis=1 stacks variables.
join="inner" keeps intersection of variables; join="outer" unions variables (may introduce missing values).
label + keys records dataset/batch provenance in .obs[label].
- Merge strategies control how conflicting
.uns and annotation columns are handled (choose based on your data governance needs).
Practical performance parameters
- Prefer sparse matrices (
csr_matrix) for count-like data.
- Convert repeated strings to categoricals (
adata.strings_to_categoricals()).
- Use compression when writing
.h5ad (e.g., compression="gzip") to reduce storage; consider zarr for chunked/cloud-friendly access.
1---2name: anndata3description: Data structure for annotated matrices in single-cell analysis; use when reading/writing .h5ad (or zarr) and exchanging data with the scverse ecosystem.4license: MIT5---6> **Source**: [https://github.com/aipoch/medical-research-skills](https://github.com/aipoch/medical-research-skills)78## When to Use910Use AnnData when you need to:1112- Load, inspect, or export annotated single-cell datasets stored as `.h5ad` (or `zarr`) for downstream tools.13- Keep a matrix (cells × features) tightly coupled with observation/feature metadata (e.g., cell types, batches, gene annotations).14- Work efficiently with large, sparse count matrices (e.g., scRNA-seq) and avoid loading everything into memory (backed mode).15- Combine multiple experiments/batches/modalities into a unified object while tracking provenance.16- Subset/filter/transform data while preserving alignment between the matrix and metadata.1718## Key Features1920- **Unified container**: `X` (data matrix) plus aligned annotations: `obs`, `var`, `uns`, and multi-dimensional slots (`obsm`, `varm`, `obsp`, `varp`), plus `layers` and optional `raw`.21- **Interoperable I/O**: Native `.h5ad` and `zarr`, plus common genomics formats (e.g., 10x, loom, mtx, csv).22- **Scalable workflows**: Sparse matrices and **backed mode** (`backed="r"`) for large datasets.23- **Safe subsetting**: Slicing preserves alignment across matrix and annotations; supports views vs copies.24- **Concatenation utilities**: `ad.concat(...)` with join/merge strategies and batch labeling; experimental lazy collections.2526> Reference notes: the original material mentions additional guides under `references/` (e.g., `references/data_structure.md`, `references/io_operations.md`, `references/concatenation.md`, `references/manipulation.md`, `references/best_practices.md`) for deeper explanations of each topic.2728## Dependencies2930- `anndata` (latest compatible with your environment; install via pip/uv)31- `numpy`32- `pandas`33- `scipy` (recommended for sparse matrices)34- Optional ecosystem tools (only if needed):35 - `scanpy`36 - `muon`37 - `torch` (for deep learning) and `anndata` experimental loader utilities3839## Example Usage4041A complete runnable example that creates an AnnData object, writes/reads `.h5ad`, subsets, concatenates batches, and demonstrates backed mode.4243```python44import numpy as np45import pandas as pd46import anndata as ad47from scipy.sparse import csr_matrix4849# ----------------------------50# 1) Create an AnnData object51# ----------------------------52rng = np.random.default_rng(0)5354n_cells, n_genes = 100, 50055X = rng.poisson(1.0, size=(n_cells, n_genes)).astype(np.float32)5657obs = pd.DataFrame(58 {59 "cell_type": (["T cell", "B cell"] * (n_cells // 2)),60 "sample": (["A", "B"] * (n_cells // 2)),61 "quality_score": rng.random(n_cells),62 },63 index=[f"cell_{i}" for i in range(n_cells)],64)6566var = pd.DataFrame(67 {"gene_name": [f"Gene_{j}" for j in range(n_genes)]},68 index=[f"ENSG{j:05d}" for j in range(n_genes)],69)7071adata = ad.AnnData(X=X, obs=obs, var=var)7273# Use sparse storage for typical count-like matrices74adata.X = csr_matrix(adata.X)7576# Convert string columns to categoricals to reduce memory and speed up ops77adata.strings_to_categoricals()7879print(f"Created: {adata.n_obs} obs × {adata.n_vars} vars")8081# ----------------------------82# 2) Write and read .h5ad83# ----------------------------84adata.write_h5ad("example.h5ad", compression="gzip")8586adata2 = ad.read_h5ad("example.h5ad")87print(f"Reloaded: {adata2.n_obs} obs × {adata2.n_vars} vars")8889# ----------------------------90# 3) Subset (keeps alignment)91# ----------------------------92t_cells = adata2[adata2.obs["cell_type"] == "T cell", :]93high_quality = adata2[adata2.obs["quality_score"] > 0.8, :]9495print(f"T cells: {t_cells.n_obs}")96print(f"High quality: {high_quality.n_obs}")9798# ----------------------------99# 4) Concatenate batches100# ----------------------------101adata_a = adata2[adata2.obs["sample"] == "A", :].copy()102adata_b = adata2[adata2.obs["sample"] == "B", :].copy()103104combined = ad.concat(105 [adata_a, adata_b],106 axis=0, # concatenate observations (cells)107 join="inner", # keep shared variables108 label="batch", # add a column in .obs109 keys=["A", "B"], # batch labels110)111112print(combined.obs["batch"].value_counts().to_dict())113114# ----------------------------115# 5) Backed mode for large files116# ----------------------------117adata_backed = ad.read_h5ad("example.h5ad", backed="r")118# Slicing in backed mode is metadata-friendly; load to memory when needed:119subset_mem = adata_backed[:10, :50].to_memory()120print(f"Backed subset loaded: {subset_mem.shape}")121```122123## Implementation Details124125### Data model (core slots)126127- `X`: primary data matrix (dense `numpy.ndarray` or sparse `scipy.sparse`), shape `(n_obs, n_vars)`.128- `obs`: per-observation metadata (`pandas.DataFrame`), indexed by `obs_names` (e.g., cell IDs).129- `var`: per-variable metadata (`pandas.DataFrame`), indexed by `var_names` (e.g., gene IDs).130- `layers`: named alternative matrices aligned to `X` (e.g., `"counts"`, `"log1p"`).131- `obsm` / `varm`: multi-dimensional embeddings aligned to obs/var (e.g., PCA, UMAP coordinates).132- `obsp` / `varp`: pairwise graphs/matrices (e.g., kNN graph in `obsp["connectivities"]`).133- `uns`: unstructured metadata (dict-like), often used for parameters and plotting configs.134- `raw` (optional): snapshot of unfiltered/untransformed data for reproducibility.135136### Views vs copies137138- Slicing like `adata_subset = adata[mask, :]` typically returns a **view** (lightweight reference).139- Use `.copy()` when you need an independent object (e.g., before in-place modifications).140141### Backed mode (large datasets)142143- `ad.read_h5ad(path, backed="r")` keeps the matrix on disk and loads data lazily.144- Convert a slice to memory with `.to_memory()` when you need in-memory computation.145- Backed mode is best for filtering by metadata, chunked processing, and avoiding OOM.146147### Concatenation behavior148149- `ad.concat([...], axis=0)` stacks observations; `axis=1` stacks variables.150- `join="inner"` keeps intersection of variables; `join="outer"` unions variables (may introduce missing values).151- `label` + `keys` records dataset/batch provenance in `.obs[label]`.152- Merge strategies control how conflicting `.uns` and annotation columns are handled (choose based on your data governance needs).153154### Practical performance parameters155156- Prefer sparse matrices (`csr_matrix`) for count-like data.157- Convert repeated strings to categoricals (`adata.strings_to_categoricals()`).158- Use compression when writing `.h5ad` (e.g., `compression="gzip"`) to reduce storage; consider `zarr` for chunked/cloud-friendly access.