Sparse Matrix Handling - Usage Guide

Sparse Matrix Handling - Usage Guide

Overview

Work with sparse matrices (CSR, CSC, COO) for memory-efficient storage and operations on count data with many zero values, especially single-cell RNA-seq data.

Prerequisites

pip install scipy pandas numpy anndata scanpy

Quick Start

Tell your AI agent what you want to do:

• "Convert my count matrix to sparse format to save memory"
• "Calculate gene means and variances on my sparse matrix"
• "Normalize my sparse single-cell counts"

Example Prompts

Conversion

"Check the sparsity of my count matrix and convert to sparse if beneficial"

"Convert my dense matrix to CSR format for row operations"

Operations

"Filter genes with fewer than 10 counts across all samples from my sparse matrix"

"Calculate CPM normalization on my sparse matrix without converting to dense"

Integration

"Create an AnnData object with my sparse counts and save as h5ad"

"Convert my sparse matrix to dense for DESeq2 export"

What the Agent Will Do

1. Assess matrix sparsity and memory footprint
2. Choose appropriate sparse format (CSR for rows, CSC for columns)
3. Convert to sparse if sparsity >50% and matrix is large
4. Perform operations while maintaining sparse format where possible
5. Convert back to dense only when required by downstream tools

When to Use Sparse

Rule: Use sparse if >50% zeros and matrix is large (>10,000 genes).

Sparse Matrix Types

import scipy.sparse as sp

csr = sp.csr_matrix(data)   # Best for: row slicing, arithmetic operations
csc = sp.csc_matrix(data)   # Best for: column slicing
coo = sp.coo_matrix(data)   # Best for: constructing sparse matrices, format conversion
lil = sp.lil_matrix((nrows, ncols))  # Best for: incrementally building matrix

Complete Workflow

import pandas as pd
import numpy as np
import scipy.sparse as sp
import anndata as ad

counts = pd.read_csv('counts.tsv', sep='\t', index_col=0)

sparsity = (counts == 0).sum().sum() / counts.size
print(f'Sparsity: {sparsity:.1%}')

if sparsity > 0.5:
    sparse_counts = sp.csr_matrix(counts.values)
    print(f'Memory: {counts.values.nbytes/1e6:.1f}MB -> {sparse_counts.data.nbytes/1e6:.1f}MB')
else:
    print('Matrix not sparse enough for conversion')

adata = ad.AnnData(
    X=sparse_counts if sparsity > 0.5 else counts.values,
    obs=pd.DataFrame(index=counts.columns),
    var=pd.DataFrame(index=counts.index)
)

adata.write_h5ad('counts.h5ad', compression='gzip')

Common Operations

Filtering

row_sums = np.array(sparse_matrix.sum(axis=1)).flatten()
keep_genes = row_sums >= 10
filtered = sparse_matrix[keep_genes, :]

col_sums = np.array(sparse_matrix.sum(axis=0)).flatten()
keep_samples = col_sums >= 1000
filtered = sparse_matrix[:, keep_samples]

Normalization

def sparse_cpm(X):
    lib_sizes = np.array(X.sum(axis=0)).flatten()
    return X.multiply(1e6 / lib_sizes)

def sparse_log1p(X):
    X = X.copy()
    X.data = np.log1p(X.data)
    return X

normalized = sparse_log1p(sparse_cpm(sparse_matrix))

Statistics

gene_means = np.array(sparse_matrix.mean(axis=1)).flatten()

def sparse_var(X):
    mean = np.array(X.mean(axis=1)).flatten()
    sq_mean = np.array(X.multiply(X).mean(axis=1)).flatten()
    return sq_mean - mean**2

gene_vars = sparse_var(sparse_matrix)

nnz_per_gene = np.array((sparse_matrix != 0).sum(axis=1)).flatten()

Integration with Analysis Tools

Scanpy

import scanpy as sc

adata = sc.read_h5ad('single_cell.h5ad')
print(f'Data type: {type(adata.X)}')

sc.pp.normalize_total(adata)
sc.pp.log1p(adata)

Converting for DESeq2/edgeR

dense_counts = sparse_matrix.toarray().astype(int)
counts_df = pd.DataFrame(dense_counts, index=gene_names, columns=sample_names)
counts_df.to_csv('counts_for_deseq.tsv', sep='\t')

Memory Management

import gc

def process_in_chunks(sparse_matrix, chunk_size=1000, func=None):
    results = []
    for i in range(0, sparse_matrix.shape[0], chunk_size):
        chunk = sparse_matrix[i:i+chunk_size, :]
        if func:
            chunk = func(chunk)
        results.append(chunk)
        gc.collect()
    return sp.vstack(results)

import psutil
print(f'Memory usage: {psutil.Process().memory_info().rss / 1e9:.1f} GB')

Troubleshooting

Out of Memory When Converting

dense = sparse_matrix.toarray()  # May fail on large matrices

result = sparse_matrix.sum(axis=0)  # Stays sparse - preferred

Slow Operations

sparse_csc = sparse_matrix.tocsc()
col_slice = sparse_csc[:, 0:10]  # Fast column slice with CSC format

Indexing Returns Matrix

val = sparse_matrix[0, 0]         # Returns matrix
val = sparse_matrix[0, 0].item()  # Returns scalar

vals = np.array(sparse_matrix[0, :].todense()).flatten()

Tips

• Use CSR format for row operations and CSC for column operations
• Avoid converting to dense unless absolutely necessary (e.g., for DESeq2)
• Process in chunks for very large matrices to manage memory
• Scanpy and AnnData handle sparse matrices natively - let them do the work
• Monitor memory usage with psutil when working with large datasets