Count Matrix Ingestion - Usage Guide

Count Matrix Ingestion - Usage Guide

Overview

Load gene expression count matrices from various quantification tools (featureCounts, Salmon, kallisto, STAR, HTSeq, 10X) into pandas DataFrames for downstream analysis.

Prerequisites

pip install pandas numpy scipy

Quick Start

Tell your AI agent what you want to do:

• "Load my featureCounts output file into a pandas DataFrame"
• "Combine Salmon quantification files from multiple samples"
• "Import STAR gene counts with reverse strand specificity"

Example Prompts

Loading Specific Formats

"Load the featureCounts file at counts.txt and clean up the sample names"

"Read all Salmon quant.sf files from the salmon_quants directory into a combined count matrix"

"Import STAR ReadsPerGene.out.tab files with reverse strandedness"

Quality Checks

"Load my count matrix and show me basic QC statistics"

"Check my count matrix for duplicate gene IDs and sum them"

Format Detection

"Figure out what delimiter my counts file uses and load it"

What the Agent Will Do

1. Identify the input format based on file structure or user specification
2. Load the data with appropriate parsing (skip headers, set index, clean sample names)
3. Run basic quality checks (shape, library sizes, zero counts, NaN values)
4. Optionally convert to sparse format for memory efficiency if matrix is highly sparse

Common Formats

Complete Loading Pipeline

import pandas as pd
import numpy as np
from pathlib import Path

class CountMatrixLoader:
    @staticmethod
    def from_featurecounts(filepath):
        df = pd.read_csv(filepath, sep='\t', comment='#')
        counts = df.set_index('Geneid').iloc[:, 5:]
        counts.columns = [c.replace('.bam', '').split('/')[-1] for c in counts.columns]
        return counts

    @staticmethod
    def from_salmon_dir(base_dir):
        base = Path(base_dir)
        samples = [d.name for d in base.iterdir() if d.is_dir() and (d / 'quant.sf').exists()]
        dfs = {}
        for sample in samples:
            sf = pd.read_csv(base / sample / 'quant.sf', sep='\t', index_col=0)
            dfs[sample] = sf['NumReads']
        return pd.DataFrame(dfs)

    @staticmethod
    def from_star_genecounts(filepaths, strandedness='reverse'):
        col_map = {'unstranded': 1, 'forward': 2, 'reverse': 3}
        col_idx = col_map[strandedness]
        dfs = {}
        for fp in filepaths:
            sample = Path(fp).name.replace('_ReadsPerGene.out.tab', '')
            df = pd.read_csv(fp, sep='\t', header=None, index_col=0)
            dfs[sample] = df.iloc[4:, col_idx - 1]
        return pd.DataFrame(dfs)

counts = CountMatrixLoader.from_featurecounts('counts.txt')
counts = CountMatrixLoader.from_salmon_dir('salmon_quants/')

Quality Checks After Loading

def check_count_matrix(counts):
    print(f'Shape: {counts.shape[0]} genes x {counts.shape[1]} samples')
    print(f'Total counts per sample:\n{counts.sum().describe()}')
    print(f'Genes with zero counts: {(counts.sum(axis=1) == 0).sum()}')
    print(f'Any NaN values: {counts.isna().any().any()}')
    print(f'Library sizes range: {counts.sum().min():.0f} - {counts.sum().max():.0f}')
    return counts

counts = check_count_matrix(counts)

Handling Different Organisms

sample_ids = counts.index[:5].tolist()
print(sample_ids)

if counts.index.str.startswith('ENSG').any():
    print('Human Ensembl gene IDs')
elif counts.index.str.startswith('ENSMUSG').any():
    print('Mouse Ensembl gene IDs')

Troubleshooting

Duplicate Gene IDs

if counts.index.duplicated().any():
    print(f'Duplicate IDs: {counts.index.duplicated().sum()}')
    counts = counts.groupby(counts.index).sum()

Missing Samples

expected = ['sample1', 'sample2', 'sample3']
actual = counts.columns.tolist()
missing = set(expected) - set(actual)
if missing:
    print(f'Missing samples: {missing}')

Wrong Delimiter

import csv
with open('counts.txt', 'r') as f:
    dialect = csv.Sniffer().sniff(f.read(1024))
    print(f'Detected delimiter: {repr(dialect.delimiter)}')

Tips

• Always check the matrix shape and library sizes after loading to catch parsing errors early
• Remove version suffixes from Ensembl IDs (e.g., ENSG00000141510.15 -> ENSG00000141510) before downstream analysis
• Use sparse matrices for single-cell data or bulk RNA-seq with >90% zeros
• Batch query multiple samples at once rather than loading one at a time for better performance