Python Bio Iterators

Iterators and Generators for Bioinformatics

When to Use

• Parsing FASTA/FASTQ files too large to fit in memory (constant memory regardless of file size)
• Generating k-mers, codons, or sliding windows lazily from long sequences or whole genomes
• Chaining multi-stage read-processing pipelines (e.g., trim → quality-filter → translate) with no intermediate lists
• Writing a custom iterator class that needs extra state a generator can't easily give you (resettable position, peek())
• Counting/aggregating over huge streams (k-mer frequency, motif positions) without materializing a list first

Version Compatibility

Python >= 3.8. Everything here is stdlib only (itertools, collections.Counter) — no third-party packages required. For production FASTA/FASTQ I/O, pair this with Biopython's SeqIO (see bio-sequence-io-read-sequences); the patterns below are for when you need custom lazy logic Biopython doesn't provide out of the box.

Prerequisites

• Comfortable with Python functions, classes, and for loops
• Basic sequence vocabulary: codons, ORFs, GC content, reading frames
• Related: bio-sequence-io-read-sequences (Biopython SeqIO.parse), bio-sequence-manipulation-codon-usage

Class-Based Iterator vs. Generator Function

Goal: iterate a DNA sequence in codon triplets, understanding when a class-based iterator is worth the extra code over a generator.

Approach: implement both. Use the class only when you need state beyond simple iteration (e.g., a resettable index or a peek() method); otherwise prefer the generator — same behavior, far less boilerplate.

class CodonIterator:
    """Iterate over a DNA sequence in codons (triplets). Use over a generator
    only if you need extra state, e.g. reset() or peek()."""

    def __init__(self, sequence):
        self.sequence = sequence
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.index + 3 > len(self.sequence):
            raise StopIteration
        codon = self.sequence[self.index:self.index + 3]
        self.index += 3
        return codon

    def reset(self):
        """Rewind to the start -- something a plain generator cannot do."""
        self.index = 0


def codon_generator(sequence):
    """Yield codons (triplets) from a DNA sequence. Preferred over the class
    above unless you specifically need reset()/peek()."""
    for i in range(0, len(sequence) - 2, 3):
        yield sequence[i:i + 3]


def kmer_generator(sequence, k):
    """Yield all overlapping k-mers of length k from a sequence, lazily."""
    for i in range(len(sequence) - k + 1):
        yield sequence[i:i + k]


if __name__ == "__main__":
    dna = "ATGAAACCCGGGTTTAAA"
    assert list(codon_generator(dna)) == [dna[i:i+3] for i in range(0, len(dna) - 2, 3)]
    ci = CodonIterator(dna)
    first_pass = list(ci)
    assert list(ci) == []          # exhausted -- single pass
    ci.reset()
    assert list(ci) == first_pass  # reset() replays it
    assert list(kmer_generator("ATGC", 2)) == ["AT", "TG", "GC"]
    print("iterator/generator equivalence OK")

Streaming FASTA/FASTQ Readers (Constant Memory)

Goal: parse arbitrarily large FASTA/FASTQ files one record at a time instead of loading the whole file into a list.

Approach: a generator that yields as it reads, plus generator stages (quality_filter, trim_n_bases) chained on top so filtering/trimming never materializes an intermediate list.

def read_fasta(filename):
    """Memory-efficient FASTA parser. Yields (header, sequence) tuples."""
    header, seq_parts = None, []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith('>'):
                if header is not None:
                    yield header, ''.join(seq_parts)
                header, seq_parts = line[1:], []
            else:
                seq_parts.append(line)
    if header is not None:
        yield header, ''.join(seq_parts)


def read_fastq(filename):
    """Streaming FASTQ reader -- yields one record dict at a time
    ('id', 'sequence', 'quality'). Memory usage is constant regardless
    of file size."""
    with open(filename) as f:
        while True:
            header = f.readline().strip()
            if not header:
                break
            sequence = f.readline().strip()
            f.readline()  # '+' separator line
            quality = f.readline().strip()
            yield {
                'id': header[1:].split()[0],
                'sequence': sequence,
                'quality': quality,
            }


def quality_filter(records, min_avg_quality=30):
    """Yield only FASTQ records with mean Phred quality >= threshold."""
    for record in records:
        avg_qual = sum(ord(c) - 33 for c in record['quality']) / len(record['quality'])
        if avg_qual >= min_avg_quality:
            yield record


def trim_n_bases(records):
    """Yield records with trailing N bases trimmed from sequence+quality."""
    for record in records:
        seq = record['sequence'].rstrip('N')
        yield {**record, 'sequence': seq, 'quality': record['quality'][:len(seq)]}


if __name__ == "__main__":
    import tempfile, os
    fastq = "@r1\nATGCNNN\n+\nIIIIIII\n@r2\nGGCC\n+\n!!!!\n"
    path = tempfile.mktemp(suffix=".fastq")
    with open(path, "w") as f:
        f.write(fastq)

    pipeline = quality_filter(trim_n_bases(read_fastq(path)), min_avg_quality=30)
    kept = list(pipeline)
    assert len(kept) == 1 and kept[0]['id'] == 'r1' and kept[0]['sequence'] == 'ATGC'
    os.remove(path)
    print("streaming FASTQ pipeline OK")

Lazy Pipelines with itertools

Goal: combine k-mer/codon streams with itertools for translation, slicing, and combinatorics without ever building a full list.

Approach: itertools.islice for lazy slicing, groupby for run-length grouping, chain/product for combining/enumerating sequences.

import itertools
from collections import Counter

CODON_TABLE = {
    'TTT': 'F', 'TTC': 'F', 'TTA': 'L', 'TTG': 'L', 'CTT': 'L', 'CTC': 'L', 'CTA': 'L', 'CTG': 'L',
    'ATT': 'I', 'ATC': 'I', 'ATA': 'I', 'ATG': 'M', 'GTT': 'V', 'GTC': 'V', 'GTA': 'V', 'GTG': 'V',
    'TCT': 'S', 'TCC': 'S', 'TCA': 'S', 'TCG': 'S', 'CCT': 'P', 'CCC': 'P', 'CCA': 'P', 'CCG': 'P',
    'ACT': 'T', 'ACC': 'T', 'ACA': 'T', 'ACG': 'T', 'GCT': 'A', 'GCC': 'A', 'GCA': 'A', 'GCG': 'A',
    'TAT': 'Y', 'TAC': 'Y', 'TAA': '*', 'TAG': '*', 'CAT': 'H', 'CAC': 'H', 'CAA': 'Q', 'CAG': 'Q',
    'AAT': 'N', 'AAC': 'N', 'AAA': 'K', 'AAG': 'K', 'GAT': 'D', 'GAC': 'D', 'GAA': 'E', 'GAG': 'E',
    'TGT': 'C', 'TGC': 'C', 'TGA': '*', 'TGG': 'W', 'CGT': 'R', 'CGC': 'R', 'CGA': 'R', 'CGG': 'R',
    'AGT': 'S', 'AGC': 'S', 'AGA': 'R', 'AGG': 'R', 'GGT': 'G', 'GGC': 'G', 'GGA': 'G', 'GGG': 'G',
}


def translate_generator(sequence):
    """Lazily translate DNA to amino acids, stopping at the first stop codon."""
    for i in range(0, len(sequence) - 2, 3):
        aa = CODON_TABLE.get(sequence[i:i + 3], 'X')
        if aa == '*':
            return
        yield aa


def all_possible_kmers(k):
    """Generator for every possible DNA k-mer of length k (4**k total)."""
    for combo in itertools.product('ATGC', repeat=k):
        yield ''.join(combo)


if __name__ == "__main__":
    dna = "ATGAAAGCCTTTGGGTGA"
    assert ''.join(translate_generator(dna)) == "MKAFG"

    # islice: take a lazy slice of an infinite/large generator without list()
    first_five = list(itertools.islice(all_possible_kmers(4), 5))
    assert len(first_five) == 5

    # groupby: homopolymer run-length detection
    runs = [(base, len(list(g))) for base, g in itertools.groupby("AAATTTGGGCC")]
    assert runs == [('A', 3), ('T', 3), ('G', 3), ('C', 2)]

    # Counter fed directly from a generator -- no intermediate list of k-mers
    kmer_counts = Counter(all_possible_kmers(2))
    assert sum(kmer_counts.values()) == 16 and len(kmer_counts) == 16

    print("itertools pipeline OK")

Pitfalls

• Single-pass exhaustion: once an iterator/generator is drained (by a for-loop or list(...)), it is empty forever — list(it) returns [] on the second call. Re-create it or materialize to a list if you need multiple passes.
• itertools functions are lazy: itertools.product(...), chain(...), etc. return iterators, not lists — wrap in list() only when you actually need to see/store all values.
• Off-by-one in k-mer/codon ranges: range(len(seq) - k + 1) for k-mers vs. range(0, len(seq) - 2, 3) for codons — mixing these up silently drops or duplicates the last window.
• Generator functions don't run until iterated: calling gen_func() executes zero lines of body code; errors inside only surface once you call next() or iterate.
• FASTQ readers assume exactly 4 lines/record: malformed or wrapped-line FASTQ (rare, but seen from some tools) will desync header/sequence/quality; validate header.startswith('@') if input is untrusted.

See Also

• bio-sequence-io-read-sequences — production FASTA/FASTQ parsing via Biopython SeqIO
• bio-sequence-io-paired-end-fastq — interleaving/pairing R1/R2 streams
• bio-sequence-manipulation-codon-usage — codon table analysis beyond simple translation
• bio-genome-intervals-interval-arithmetic — lazy interval/window operations over genomic coordinates