Python Bio File Operations

File Operations for Bioinformatics

When to Use

• Parsing a FASTA/FASTQ file into headers and sequences without pulling in BioPython.
• Writing sequences to FASTA with proper line wrapping (60/70/80 chars).
• Reading/writing gene expression tables (CSV) or genomic intervals (BED/TSV) with csv.DictReader/DictWriter.
• Streaming a multi-GB FASTA/FASTQ file line-by-line instead of loading it whole.
• Serializing intermediate analysis results (dicts with sets/numpy arrays) with pickle, or structured records with json.

Version Compatibility

Pure standard library — open(), csv, json, pickle, gzip — works unchanged on Python ≥3.8 (examples use f-strings and csv.DictReader, available since 3.6+).

Prerequisites

• No third-party packages required (stdlib only: csv, json, pickle, gzip).
• For heavier FASTA/FASTQ/SAM work, see bio-sequence-io-read-sequences and biopython — this skill covers hand-rolled parsers for when a dependency isn't warranted or you need full control over streaming.

Pitfalls

• strip() when reading lines: every line from a file has a trailing \n. Forgetting this causes sequences to contain invisible newlines that break string comparisons and length calculations.
• "w" mode destroys existing content. Use "a" to append, "x" to fail loudly if the file already exists.
• Binary mode for BAM/gzip/pickle: open(path, "rb")/"wb". Text mode tries to decode bytes as UTF-8 and corrupts binary data (or raises UnicodeDecodeError).
• f.read() and f.readlines() load the entire file into memory. For large FASTA/FASTQ, iterate with for line in f: or write a generator — memory stays constant regardless of file size.
• Forgetting the last record. Streaming FASTA/FASTQ parsers accumulate a sequence until they see the next header (or EOF) — the final record must be flushed explicitly after the loop ends.
• csv.writer/DictWriter on Windows/text mode: always open with newline='' when writing CSV, otherwise rows get doubled \r\n\r\n line endings.
• set objects are not JSON-serializable. json.dump({'x': {1,2}}) raises TypeError; use pickle for arbitrary Python objects (sets, numpy arrays, custom classes), json only for portable dict/list/str/number data.

Reading Methods

FASTA Parser (streaming)

Goal: turn a FASTA file into {header: sequence} without loading it as one giant string. Approach: iterate line-by-line, accumulate sequence chunks in a list (fast ''.join), flush on each new > header and again after the loop for the final record.

def read_fasta(filename):
    """Parse a FASTA file and return a dict mapping full headers to sequences.

    Args:
        filename: path to a FASTA file.

    Returns:
        dict of {header (without '>'): concatenated sequence string}.
    """
    sequences = {}
    current_header = None
    current_seq = []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith('>'):
                if current_header is not None:
                    sequences[current_header] = ''.join(current_seq)
                current_header = line[1:]
                current_seq = []
            else:
                current_seq.append(line)
        if current_header is not None:  # don't forget the last sequence!
            sequences[current_header] = ''.join(current_seq)
    return sequences

FASTA Writer (with line wrapping) and Generator Parser

Goal: write sequences back out with wrapped lines, and parse huge FASTA files with constant memory via a generator. Approach: write_fasta accepts a dict or a list of (header, seq) tuples; the generator yields one record at a time instead of building a dict, so callers can for header, seq in parse_fasta_generator(path): over a multi-GB file.

def write_fasta(sequences, filename, line_width=60):
    """Write sequences (dict or list of (header, seq) tuples) to FASTA, wrapped at line_width."""
    items = sequences.items() if isinstance(sequences, dict) else sequences
    with open(filename, 'w') as f:
        for header, seq in items:
            f.write(f">{header}\n")
            for i in range(0, len(seq), line_width):
                f.write(seq[i:i + line_width] + '\n')


def parse_fasta_generator(filename):
    """Yield (header, sequence) tuples one at a time — constant memory regardless of file size."""
    current_header = None
    current_seq = []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith('>'):
                if current_header is not None:
                    yield current_header, ''.join(current_seq)
                current_header = line[1:]
                current_seq = []
            else:
                current_seq.append(line)
        if current_header is not None:
            yield current_header, ''.join(current_seq)

CSV/TSV, JSON, and Pickle

Goal: read/write tabular gene-expression and BED-style data by column name, plus serialize structured or arbitrary Python results. Approach: csv.DictReader/DictWriter for row-as-dict access (delimiter='\t' for BED/TSV); json.dump/load for portable records; pickle.dump/load (binary mode) for objects json can't handle, like sets or numpy arrays.

import csv
import json
import pickle

# Read CSV with column-name access
with open('gene_expression.csv') as f:
    for row in csv.DictReader(f):
        print(row['gene_name'], float(row['expression_sample1']))

# Read TSV / BED files the same way
with open('genes.bed') as f:
    for row in csv.DictReader(f, delimiter='\t'):
        print(f"{row['gene']}: {row['chromosome']}:{row['start']}-{row['end']}")

# Write CSV with a header row (newline='' avoids blank rows on Windows)
with open('de_results.csv', 'w', newline='') as f:
    writer = csv.DictWriter(f, fieldnames=['gene', 'fold_change', 'p_value'])
    writer.writeheader()
    writer.writerows(results)  # list of dicts with those keys

# JSON round-trip for portable structured data
with open('gene.json', 'w') as f:
    json.dump(gene_annotation, f, indent=2)
with open('gene.json') as f:
    data = json.load(f)

# Pickle for arbitrary Python objects (sets, numpy arrays, custom classes)
with open('results.pkl', 'wb') as f:
    pickle.dump(analysis_results, f)
with open('results.pkl', 'rb') as f:
    loaded = pickle.load(f)

See Also

• bio-sequence-io-read-sequences — BioPython SeqIO for FASTA/FASTQ/GenBank parsing when you don't want a hand-rolled parser.
• bio-sequence-io-compressed-files — same patterns over .gz-compressed inputs via gzip.open.
• python-bio-context-managers — the with statement mechanics these examples rely on.
• python-bio-generators — more on yield-based streaming parsers like parse_fasta_generator.