# Python Bio Iterators

> Stream FASTA/FASTQ and generate k-mers/codons lazily with Python generators, custom __iter__/__next__ classes, and itertools. Use when parsing multi-GB sequence files without loading them fully into RAM or chaining filter-trim-translate pipelines.

- Skill: `pavel-kravchenko/python-bio-iterators` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pavel-kravchenko/python-bio-iterators`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pavel-kravchenko/python-bio-iterators/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: pavel-kravchenko (https://skillmd.com/u/pavel-kravchenko)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pavel-kravchenko/python-bio-iterators

---


# Iterators and Generators for Bioinformatics

## When to Use

- Parsing FASTA/FASTQ files too large to fit in memory (constant memory regardless of file size)
- Generating k-mers, codons, or sliding windows lazily from long sequences or whole genomes
- Chaining multi-stage read-processing pipelines (e.g., trim → quality-filter → translate) with no intermediate lists
- Writing a custom iterator class that needs extra state a generator can't easily give you (resettable position, `peek()`)
- Counting/aggregating over huge streams (k-mer frequency, motif positions) without materializing a list first

## Version Compatibility

Python >= 3.8. Everything here is stdlib only (`itertools`, `collections.Counter`) — no third-party packages required. For production FASTA/FASTQ I/O, pair this with Biopython's `SeqIO` (see `bio-sequence-io-read-sequences`); the patterns below are for when you need custom lazy logic Biopython doesn't provide out of the box.

## Prerequisites

- Comfortable with Python functions, classes, and `for` loops
- Basic sequence vocabulary: codons, ORFs, GC content, reading frames
- Related: `bio-sequence-io-read-sequences` (Biopython `SeqIO.parse`), `bio-sequence-manipulation-codon-usage`

## Class-Based Iterator vs. Generator Function

**Goal:** iterate a DNA sequence in codon triplets, understanding when a class-based iterator is worth the extra code over a generator.

**Approach:** implement both. Use the class only when you need state beyond simple iteration (e.g., a resettable index or a `peek()` method); otherwise prefer the generator — same behavior, far less boilerplate.

```python
class CodonIterator:
    """Iterate over a DNA sequence in codons (triplets). Use over a generator
    only if you need extra state, e.g. reset() or peek()."""

    def __init__(self, sequence):
        self.sequence = sequence
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.index + 3 > len(self.sequence):
            raise StopIteration
        codon = self.sequence[self.index:self.index + 3]
        self.index += 3
        return codon

    def reset(self):
        """Rewind to the start -- something a plain generator cannot do."""
        self.index = 0


def codon_generator(sequence):
    """Yield codons (triplets) from a DNA sequence. Preferred over the class
    above unless you specifically need reset()/peek()."""
    for i in range(0, len(sequence) - 2, 3):
        yield sequence[i:i + 3]


def kmer_generator(sequence, k):
    """Yield all overlapping k-mers of length k from a sequence, lazily."""
    for i in range(len(sequence) - k + 1):
        yield sequence[i:i + k]


if __name__ == "__main__":
    dna = "ATGAAACCCGGGTTTAAA"
    assert list(codon_generator(dna)) == [dna[i:i+3] for i in range(0, len(dna) - 2, 3)]
    ci = CodonIterator(dna)
    first_pass = list(ci)
    assert list(ci) == []          # exhausted -- single pass
    ci.reset()
    assert list(ci) == first_pass  # reset() replays it
    assert list(kmer_generator("ATGC", 2)) == ["AT", "TG", "GC"]
    print("iterator/generator equivalence OK")
```

## Streaming FASTA/FASTQ Readers (Constant Memory)

**Goal:** parse arbitrarily large FASTA/FASTQ files one record at a time instead of loading the whole file into a list.

**Approach:** a generator that yields as it reads, plus generator stages (`quality_filter`, `trim_n_bases`) chained on top so filtering/trimming never materializes an intermediate list.

```python
def read_fasta(filename):
    """Memory-efficient FASTA parser. Yields (header, sequence) tuples."""
    header, seq_parts = None, []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith('>'):
                if header is not None:
                    yield header, ''.join(seq_parts)
                header, seq_parts = line[1:], []
            else:
                seq_parts.append(line)
    if header is not None:
        yield header, ''.join(seq_parts)


def read_fastq(filename):
    """Streaming FASTQ reader -- yields one record dict at a time
    ('id', 'sequence', 'quality'). Memory usage is constant regardless
    of file size."""
    with open(filename) as f:
        while True:
            header = f.readline().strip()
            if not header:
                break
            sequence = f.readline().strip()
            f.readline()  # '+' separator line
            quality = f.readline().strip()
            yield {
                'id': header[1:].split()[0],
                'sequence': sequence,
                'quality': quality,
            }


def quality_filter(records, min_avg_quality=30):
    """Yield only FASTQ records with mean Phred quality >= threshold."""
    for record in records:
        avg_qual = sum(ord(c) - 33 for c in record['quality']) / len(record['quality'])
        if avg_qual >= min_avg_quality:
            yield record


def trim_n_bases(records):
    """Yield records with trailing N bases trimmed from sequence+quality."""
    for record in records:
        seq = record['sequence'].rstrip('N')
        yield {**record, 'sequence': seq, 'quality': record['quality'][:len(seq)]}


if __name__ == "__main__":
    import tempfile, os
    fastq = "@r1\nATGCNNN\n+\nIIIIIII\n@r2\nGGCC\n+\n!!!!\n"
    path = tempfile.mktemp(suffix=".fastq")
    with open(path, "w") as f:
        f.write(fastq)

    pipeline = quality_filter(trim_n_bases(read_fastq(path)), min_avg_quality=30)
    kept = list(pipeline)
    assert len(kept) == 1 and kept[0]['id'] == 'r1' and kept[0]['sequence'] == 'ATGC'
    os.remove(path)
    print("streaming FASTQ pipeline OK")
```

## Lazy Pipelines with `itertools`

**Goal:** combine k-mer/codon streams with `itertools` for translation, slicing, and combinatorics without ever building a full list.

**Approach:** `itertools.islice` for lazy slicing, `groupby` for run-length grouping, `chain`/`product` for combining/enumerating sequences.

```python
import itertools
from collections import Counter

CODON_TABLE = {
    'TTT': 'F', 'TTC': 'F', 'TTA': 'L', 'TTG': 'L', 'CTT': 'L', 'CTC': 'L', 'CTA': 'L', 'CTG': 'L',
    'ATT': 'I', 'ATC': 'I', 'ATA': 'I', 'ATG': 'M', 'GTT': 'V', 'GTC': 'V', 'GTA': 'V', 'GTG': 'V',
    'TCT': 'S', 'TCC': 'S', 'TCA': 'S', 'TCG': 'S', 'CCT': 'P', 'CCC': 'P', 'CCA': 'P', 'CCG': 'P',
    'ACT': 'T', 'ACC': 'T', 'ACA': 'T', 'ACG': 'T', 'GCT': 'A', 'GCC': 'A', 'GCA': 'A', 'GCG': 'A',
    'TAT': 'Y', 'TAC': 'Y', 'TAA': '*', 'TAG': '*', 'CAT': 'H', 'CAC': 'H', 'CAA': 'Q', 'CAG': 'Q',
    'AAT': 'N', 'AAC': 'N', 'AAA': 'K', 'AAG': 'K', 'GAT': 'D', 'GAC': 'D', 'GAA': 'E', 'GAG': 'E',
    'TGT': 'C', 'TGC': 'C', 'TGA': '*', 'TGG': 'W', 'CGT': 'R', 'CGC': 'R', 'CGA': 'R', 'CGG': 'R',
    'AGT': 'S', 'AGC': 'S', 'AGA': 'R', 'AGG': 'R', 'GGT': 'G', 'GGC': 'G', 'GGA': 'G', 'GGG': 'G',
}


def translate_generator(sequence):
    """Lazily translate DNA to amino acids, stopping at the first stop codon."""
    for i in range(0, len(sequence) - 2, 3):
        aa = CODON_TABLE.get(sequence[i:i + 3], 'X')
        if aa == '*':
            return
        yield aa


def all_possible_kmers(k):
    """Generator for every possible DNA k-mer of length k (4**k total)."""
    for combo in itertools.product('ATGC', repeat=k):
        yield ''.join(combo)


if __name__ == "__main__":
    dna = "ATGAAAGCCTTTGGGTGA"
    assert ''.join(translate_generator(dna)) == "MKAFG"

    # islice: take a lazy slice of an infinite/large generator without list()
    first_five = list(itertools.islice(all_possible_kmers(4), 5))
    assert len(first_five) == 5

    # groupby: homopolymer run-length detection
    runs = [(base, len(list(g))) for base, g in itertools.groupby("AAATTTGGGCC")]
    assert runs == [('A', 3), ('T', 3), ('G', 3), ('C', 2)]

    # Counter fed directly from a generator -- no intermediate list of k-mers
    kmer_counts = Counter(all_possible_kmers(2))
    assert sum(kmer_counts.values()) == 16 and len(kmer_counts) == 16

    print("itertools pipeline OK")
```

## Pitfalls

- **Single-pass exhaustion:** once an iterator/generator is drained (by a for-loop or `list(...)`), it is empty forever — `list(it)` returns `[]` on the second call. Re-create it or materialize to a list if you need multiple passes.
- **`itertools` functions are lazy:** `itertools.product(...)`, `chain(...)`, etc. return iterators, not lists — wrap in `list()` only when you actually need to see/store all values.
- **Off-by-one in k-mer/codon ranges:** `range(len(seq) - k + 1)` for k-mers vs. `range(0, len(seq) - 2, 3)` for codons — mixing these up silently drops or duplicates the last window.
- **Generator functions don't run until iterated:** calling `gen_func()` executes zero lines of body code; errors inside only surface once you call `next()` or iterate.
- **FASTQ readers assume exactly 4 lines/record:** malformed or wrapped-line FASTQ (rare, but seen from some tools) will desync `header`/`sequence`/`quality`; validate `header.startswith('@')` if input is untrusted.

## See Also

- `bio-sequence-io-read-sequences` — production FASTA/FASTQ parsing via Biopython `SeqIO`
- `bio-sequence-io-paired-end-fastq` — interleaving/pairing R1/R2 streams
- `bio-sequence-manipulation-codon-usage` — codon table analysis beyond simple translation
- `bio-genome-intervals-interval-arithmetic` — lazy interval/window operations over genomic coordinates