# Python Bio File Operations

> Read/write FASTA, FASTQ, CSV/TSV (BED), JSON, and pickle files in Python using open()/context managers, csv.DictReader/DictWriter, and streaming generators for large genomics files. Use when parsing a FASTA/FASTQ file, writing sequences back out with line wrapping, reading/writing gene expression CSV or BED/TSV files, loading GenBank ORIGIN sequence, saving Python objects with pickle, or handling files too large to fit in memory.

- Skill: `pavel-kravchenko/python-bio-file-operations` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pavel-kravchenko/python-bio-file-operations`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pavel-kravchenko/python-bio-file-operations/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: pavel-kravchenko (https://skillmd.com/u/pavel-kravchenko)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pavel-kravchenko/python-bio-file-operations

---


# File Operations for Bioinformatics

## When to Use

- Parsing a FASTA/FASTQ file into headers and sequences without pulling in BioPython.
- Writing sequences to FASTA with proper line wrapping (60/70/80 chars).
- Reading/writing gene expression tables (CSV) or genomic intervals (BED/TSV) with `csv.DictReader`/`DictWriter`.
- Streaming a multi-GB FASTA/FASTQ file line-by-line instead of loading it whole.
- Serializing intermediate analysis results (dicts with sets/numpy arrays) with `pickle`, or structured records with `json`.

## Version Compatibility

Pure standard library — `open()`, `csv`, `json`, `pickle`, `gzip` — works unchanged on Python ≥3.8 (examples use f-strings and `csv.DictReader`, available since 3.6+).

## Prerequisites

- No third-party packages required (stdlib only: `csv`, `json`, `pickle`, `gzip`).
- For heavier FASTA/FASTQ/SAM work, see `bio-sequence-io-read-sequences` and `biopython` — this skill covers hand-rolled parsers for when a dependency isn't warranted or you need full control over streaming.

## Pitfalls

- **`strip()` when reading lines:** every line from a file has a trailing `\n`. Forgetting this causes sequences to contain invisible newlines that break string comparisons and length calculations.
- **`"w"` mode destroys existing content.** Use `"a"` to append, `"x"` to fail loudly if the file already exists.
- **Binary mode for BAM/gzip/pickle:** `open(path, "rb")`/`"wb"`. Text mode tries to decode bytes as UTF-8 and corrupts binary data (or raises `UnicodeDecodeError`).
- **`f.read()` and `f.readlines()` load the entire file into memory.** For large FASTA/FASTQ, iterate with `for line in f:` or write a generator — memory stays constant regardless of file size.
- **Forgetting the last record.** Streaming FASTA/FASTQ parsers accumulate a sequence until they see the *next* header (or EOF) — the final record must be flushed explicitly after the loop ends.
- **`csv.writer`/`DictWriter` on Windows/text mode:** always open with `newline=''` when writing CSV, otherwise rows get doubled `\r\n\r\n` line endings.
- **`set` objects are not JSON-serializable.** `json.dump({'x': {1,2}})` raises `TypeError`; use `pickle` for arbitrary Python objects (sets, numpy arrays, custom classes), `json` only for portable dict/list/str/number data.

## Reading Methods

| Method | Memory | Use when |
|--------|--------|----------|
| `f.read()` | Loads entire file | Small file, need the whole string at once |
| `f.readline()` | One line | Need to peek at just the first line(s) |
| `f.readlines()` | Loads entire file (as list) | Small file, need random access to lines |
| `for line in f:` | One line at a time (preferred) | Large files, streaming/generator parsers |

## FASTA Parser (streaming)

**Goal:** turn a FASTA file into `{header: sequence}` without loading it as one giant string.
**Approach:** iterate line-by-line, accumulate sequence chunks in a list (fast `''.join`), flush on each new `>` header and again after the loop for the final record.

```python
def read_fasta(filename):
    """Parse a FASTA file and return a dict mapping full headers to sequences.

    Args:
        filename: path to a FASTA file.

    Returns:
        dict of {header (without '>'): concatenated sequence string}.
    """
    sequences = {}
    current_header = None
    current_seq = []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith('>'):
                if current_header is not None:
                    sequences[current_header] = ''.join(current_seq)
                current_header = line[1:]
                current_seq = []
            else:
                current_seq.append(line)
        if current_header is not None:  # don't forget the last sequence!
            sequences[current_header] = ''.join(current_seq)
    return sequences
```

## FASTA Writer (with line wrapping) and Generator Parser

**Goal:** write sequences back out with wrapped lines, and parse huge FASTA files with constant memory via a generator.
**Approach:** `write_fasta` accepts a dict or a list of `(header, seq)` tuples; the generator yields one record at a time instead of building a dict, so callers can `for header, seq in parse_fasta_generator(path):` over a multi-GB file.

```python
def write_fasta(sequences, filename, line_width=60):
    """Write sequences (dict or list of (header, seq) tuples) to FASTA, wrapped at line_width."""
    items = sequences.items() if isinstance(sequences, dict) else sequences
    with open(filename, 'w') as f:
        for header, seq in items:
            f.write(f">{header}\n")
            for i in range(0, len(seq), line_width):
                f.write(seq[i:i + line_width] + '\n')


def parse_fasta_generator(filename):
    """Yield (header, sequence) tuples one at a time — constant memory regardless of file size."""
    current_header = None
    current_seq = []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith('>'):
                if current_header is not None:
                    yield current_header, ''.join(current_seq)
                current_header = line[1:]
                current_seq = []
            else:
                current_seq.append(line)
        if current_header is not None:
            yield current_header, ''.join(current_seq)
```

## CSV/TSV, JSON, and Pickle

**Goal:** read/write tabular gene-expression and BED-style data by column name, plus serialize structured or arbitrary Python results.
**Approach:** `csv.DictReader`/`DictWriter` for row-as-dict access (`delimiter='\t'` for BED/TSV); `json.dump`/`load` for portable records; `pickle.dump`/`load` (binary mode) for objects `json` can't handle, like sets or numpy arrays.

```python
import csv
import json
import pickle

# Read CSV with column-name access
with open('gene_expression.csv') as f:
    for row in csv.DictReader(f):
        print(row['gene_name'], float(row['expression_sample1']))

# Read TSV / BED files the same way
with open('genes.bed') as f:
    for row in csv.DictReader(f, delimiter='\t'):
        print(f"{row['gene']}: {row['chromosome']}:{row['start']}-{row['end']}")

# Write CSV with a header row (newline='' avoids blank rows on Windows)
with open('de_results.csv', 'w', newline='') as f:
    writer = csv.DictWriter(f, fieldnames=['gene', 'fold_change', 'p_value'])
    writer.writeheader()
    writer.writerows(results)  # list of dicts with those keys

# JSON round-trip for portable structured data
with open('gene.json', 'w') as f:
    json.dump(gene_annotation, f, indent=2)
with open('gene.json') as f:
    data = json.load(f)

# Pickle for arbitrary Python objects (sets, numpy arrays, custom classes)
with open('results.pkl', 'wb') as f:
    pickle.dump(analysis_results, f)
with open('results.pkl', 'rb') as f:
    loaded = pickle.load(f)
```

## See Also

- `bio-sequence-io-read-sequences` — BioPython `SeqIO` for FASTA/FASTQ/GenBank parsing when you don't want a hand-rolled parser.
- `bio-sequence-io-compressed-files` — same patterns over `.gz`-compressed inputs via `gzip.open`.
- `python-bio-context-managers` — the `with` statement mechanics these examples rely on.
- `python-bio-generators` — more on `yield`-based streaming parsers like `parse_fasta_generator`.