HolobiomicsLab anndata-backed-object-manipulation
Summary
Manipulate fully backed AnnData objects to store and access single-cell omics fragment data and count matrices without loading entire datasets into memory. This skill enables scalable preprocessing and matrix operations on datasets exceeding 10 million cells by leveraging on-disk storage.
When to use
When working with large single-cell ATAC-seq or multi-omics datasets where in-memory storage is infeasible (>1M cells), and you need to iteratively add or modify count matrices (tile-based, peak-based, or gene-based) while preserving fragment-level data for reproducibility and re-analysis.
When NOT to use
- Input data is already a dense in-memory count matrix; use backed AnnData only to avoid reprocessing raw fragments.
- Workflow requires frequent random access to all cells × features values (backed mode has I/O latency); consider in-memory AnnData for small datasets (<1M cells).
- Fragment data is unavailable or lost; count matrices cannot be regenerated with alternative binning strategies without raw sequencing alignments.
Inputs
- backed AnnData object (.h5ad file on disk)
- fragment coordinate data stored in .obsm slots (paired-end or single-end fragment tuples)
- genomic interval definitions (tile coordinates, peak BED file, or gene GTF annotations)
Outputs
- count matrix (stored in .X or .obsm of the same backed AnnData object)
- updated backed AnnData object with added matrix layer
- metadata about matrix binning (tile size, peak list, gene annotations)
How to apply
Load or create a backed AnnData object using SnapATAC2's I/O functions, which stores dense arrays and sparse matrices on disk rather than in RAM. Store raw fragment coordinates in .obsm['fragment_paired'] or .obsm['fragment_single'] slots. Apply matrix operations (pp.add_tile_matrix, pp.make_peak_matrix, pp.make_gene_matrix) to generate count matrices from fragments; each operation reads from and writes to disk without materializing the full dataset. Verify matrix shape (n_obs × n_vars) and sparsity after each operation to confirm the count matrix reflects the intended genomic binning or feature selection.
Related tools
- SnapATAC2 (Provides pp.add_tile_matrix, pp.make_peak_matrix, pp.make_gene_matrix functions for matrix operations on backed AnnData; handles fragment data I/O and count aggregation) — https://github.com/scverse/SnapATAC2
- precellar (Upstream preprocessing: converts raw FASTQ files to fragment files (.tsv.zst) and count matrices that are input to backed AnnData workflows) — https://github.com/regulatory-genomics/precellar
- Scanpy (Companion library for seamless integration with backed AnnData objects; provides embedding and clustering on top of SnapATAC2 matrices)
Examples
import snapatac2 as snap; adata = snap.read('data.h5ad', backed='r+'); snap.pp.add_tile_matrix(adata, counting_strategy='paired_insertion'); adata.write()
Evaluation signals
- Verify that the backed AnnData object file size on disk grows incrementally with each matrix operation; no explosion to in-memory size.
- Check that .X (or intended .obsm key) contains the expected count matrix shape: n_obs = number of cells, n_vars = number of tiles/peaks/genes.
- Confirm matrix is sparse (majority zeros) and contains non-zero entries distributed across cells and genomic coordinates; spot-check a few cell × feature pairs for plausible counts.
- Run pp.filter_cells or pp.select_features on the matrix output to verify that downstream filtering operations read and write correctly to the backed object.
- Profile disk I/O latency: operations should complete in reasonable time (seconds to minutes per matrix depending on dataset size); significant slowdown may indicate backing strategy issues.
Limitations
- Backed AnnData objects require consistent on-disk storage; file system failures or accidental deletion of .h5ad file causes data loss.
- Matrix operations are sequential and I/O-bound; random-access patterns (e.g., selecting arbitrary subsets of cells repeatedly) may be slower than in-memory alternatives.
- Fragment storage in .obsm as tuples or coordinate arrays can itself consume significant disk space for large datasets; compression (e.g., via precellar's .tsv.zst) is recommended upstream.
- Compatibility with older AnnData or SnapATAC2 versions may break if the backing format or .obsm schema changes.
Evidence
- [readme] Implementation of fully backed AnnData.: "Implementation of fully backed AnnData."
- [other] Matrix operation including pp.add_tile_matrix, pp.make_peak_matrix, pp.make_gene_matrix: "Matrix operation including pp.add_tile_matrix, pp.make_peak_matrix, pp.make_gene_matrix"
- [other] Load backed AnnData object containing fragment data (stored in .obsm['fragment_paired'] or .obsm['fragment_single']). Invoke pp.add_tile_matrix with counting_strategy='paired_insertion': "Load backed AnnData object containing fragment data (stored in .obsm['fragment_paired'] or .obsm['fragment_single']). Invoke pp.add_tile_matrix with counting_strategy='paired_insertion'"
- [readme] Scale to more than 10 million cells.: "Scale to more than 10 million cells."
- [readme] Blazingly fast preprocessing tools for BAM to fragment files conversion and count matrix generation.: "Blazingly fast preprocessing tools for BAM to fragment files conversion and count matrix generation."