# Foundations Character Encodings

> Fix UTF-8/Latin-1/CP1251 decode errors, strip BOM, repair mojibake, normalize CRLF endings, remove hidden Unicode in FASTA/GFF text. Use when hitting UnicodeDecodeError or garbled text from Windows/Excel exports.

- Skill: `pavel-kravchenko/foundations-character-encodings` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pavel-kravchenko/foundations-character-encodings`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pavel-kravchenko/foundations-character-encodings/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: pavel-kravchenko (https://skillmd.com/u/pavel-kravchenko)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pavel-kravchenko/foundations-character-encodings

---


# Character Encodings and Binary Data in Bioinformatics

## When to Use

- `UnicodeDecodeError: 'utf-8' codec can't decode byte 0x...` when opening a file downloaded from an external database or exported from Windows.
- FASTA/GFF/GenBank parsing gives sequences that are one character too long, or headers that start with an invisible character.
- Text copy-pasted from a paper or Word document (author names, organism names, Greek letters like α/β, µ, °) looks garbled after being read into Python.
- A gene/protein name with special characters (`HLA-A*02:01`, `C/EBPα`) needs to go into a URL query (UniProt, NCBI E-utilities).
- Need to detect or convert a file's encoding (e.g., CP1251 Cyrillic annotations) to UTF-8 before downstream tools choke on it.

## Version Compatibility

Python ≥3.9 (all examples use only the standard library: `codecs`/`str.encode`/`str.decode`, `unicodedata`, `urllib.parse`). Optional: `chardet` ≥5.0 for encoding auto-detection.

## Prerequisites

- Basic Python string/bytes distinction (`str` vs `bytes`, `.encode()`/`.decode()`).
- `pip install chardet` (optional, only needed for auto-detection).

## From Bits to Bytes

| Notation | Example | Meaning |
|----------|---------|---------|
| Binary | `01000001` | 8 bits = 1 byte |
| Decimal | `65` | Human-friendly |
| Hexadecimal | `0x41` | 2 hex digits = 1 byte |

```python
for value in [0, 10, 32, 48, 65, 90, 97, 122, 127, 200, 255]:
    char = chr(value) if 32 <= value < 127 else '--'
    print(f"{value:7d}  {format(value, '08b'):>10}  0x{format(value, '02X')}  '{char}'")
```

**Bioinformatics relevance:** DNA (`ATGCN`), RNA (`AUGC`), and amino acid codes are pure ASCII (byte values 0-127) -- sequence data itself never has encoding issues. Problems arise in annotations, author names, organism names, and free-text fields, and in FASTQ quality strings (Phred+33 maps quality 0-93 to ASCII 33-126).

```python
# FASTQ quality scores: Phred+33 maps quality 0-93 to ASCII 33-126
quality_string = "IIIIIIIIFFFFFFDDDDDBBBB"
for char in quality_string[:6]:
    phred = ord(char) - 33
    print(f"'{char}' -> Phred {phred} -> error rate {10 ** (-phred / 10):.6f}")
```

## Core Operations

**Goal:** detect why a file fails to decode, repair mojibake, and normalize the text before parsing.
**Approach:** try encodings in priority order (UTF-8 with BOM check first, Latin-1 last since it never raises), fix already-mis-decoded text by round-tripping through the wrong encoding, and strip invisible Unicode noise from biological sequences.

```python
def read_text_file(filepath):
    """Read a text file, trying common bioinformatics encodings in order.

    Priority:
    1. utf-8-sig  (UTF-8, auto-strips a BOM if a Windows editor added one)
    2. utf-8      (modern standard, no BOM)
    3. latin-1    (never fails -- last-resort fallback, may give wrong characters)
    """
    for encoding in ('utf-8-sig', 'utf-8', 'latin-1'):
        try:
            with open(filepath, 'r', encoding=encoding) as f:
                content = f.read()
            if encoding == 'latin-1':
                print(f"WARNING: fell back to Latin-1 for {filepath}; "
                      "some characters may be wrong -- consider converting to UTF-8.")
            return content
        except UnicodeDecodeError:
            continue
    raise ValueError(f"Could not decode {filepath} with any known encoding")


def fix_mojibake(text, wrong_encoding='latin-1', right_encoding='utf-8'):
    """Repair text that was UTF-8 but got decoded as Latin-1 (classic mojibake).

    Round-trips through the wrong encoding's byte representation, then
    decodes those bytes with the correct encoding.
    """
    return text.encode(wrong_encoding).decode(right_encoding)


# Example: alpha-helix mis-decoded, then repaired
original = "α-helix structure"
utf8_bytes = original.encode('utf-8')
wrong = utf8_bytes.decode('latin-1')          # garbled, but does not raise
fixed = fix_mojibake(wrong)
assert fixed == original
```

```python
import unicodedata


def sanitize_sequence(seq, valid_chars='ATGCNatgcn'):
    """Strip characters that are not valid sequence letters or expected whitespace.

    Catches copy-paste artifacts like zero-width space (U+200B) and
    non-breaking space (U+00A0) that look identical to normal text but
    break equality checks, split(), and length calculations.
    """
    valid_set = set(valid_chars)
    cleaned, removed = [], []
    for char in seq:
        if char in valid_set:
            cleaned.append(char)
        elif char not in ('\n', '\r', ' ', '\t'):
            removed.append(f"'{char}' ({unicodedata.name(char, f'U+{ord(char):04X}')})")
    if removed:
        print(f"WARNING: removed {len(removed)} unexpected characters: {', '.join(removed)}")
    return ''.join(cleaned)


messy = "ATG​CGA TCGA\r\nGGG­TTT"
print(sanitize_sequence(messy))  # -> ATGCGATCGAGGGTTT
```

```python
def parse_fasta_robust(raw_bytes):
    """Parse FASTA from raw bytes, handling BOM and any line-ending style."""
    text = raw_bytes.decode('utf-8-sig').replace('\r\n', '\n').replace('\r', '\n')
    sequences, current_header, current_seq = {}, None, []
    for line in text.strip().split('\n'):
        line = line.strip()
        if line.startswith('>'):
            if current_header is not None:
                sequences[current_header] = ''.join(current_seq)
            current_header, current_seq = line[1:], []
        elif line:
            current_seq.append(line)
    if current_header is not None:
        sequences[current_header] = ''.join(current_seq)
    return sequences


windows_fasta = b">gene1\r\nATGCGATCGA\r\nTTTAAAGGGC\r\n"
result = parse_fasta_robust(windows_fasta)
assert result['gene1'] == 'ATGCGATCGATTTAAAGGGC'
```

**Goal:** detect an unknown encoding, and percent-encode gene/protein names for database URLs.
**Approach:** use `chardet` when the source encoding is unknown; use `urllib.parse.quote`/`unquote` for E-utilities/UniProt query strings containing `*`, `/`, `+`, or Greek letters.

```python
from urllib.parse import quote, unquote

try:
    import chardet

    def detect_encoding(raw_bytes):
        """Return chardet's best-guess encoding and confidence for raw bytes."""
        result = chardet.detect(raw_bytes)
        return result['encoding'], result['confidence']

    guess, conf = detect_encoding("Анализ экспрессии генов".encode('cp1251'))
    print(f"detected {guess} (confidence {conf:.0%})")
except ImportError:
    print("chardet not installed: pip install chardet")

# HLA allele / transcription factor names in a UniProt/NCBI query URL
gene_queries = ["HLA-A*02:01", "C/EBPα", "Na+/K+ ATPase"]
base_url = "https://www.uniprot.org/uniprot/?query="
for gene in gene_queries:
    print(f"{gene:<15} -> {base_url}{quote(gene, safe='')}")

encoded_url = "https://www.ncbi.nlm.nih.gov/gene/?term=C%2FEBP%CE%B1"
print(unquote(encoded_url))  # -> .../gene/?term=C/EBPα
```

## Pitfalls

- **Latin-1 never fails -- but that does not mean it is right**: Latin-1 decodes every byte 0-255 successfully, silently producing wrong characters (mojibake) when the file is actually UTF-8 with multi-byte sequences. Never use it as a first choice, only as a documented fallback.
- **UTF-8 BOM confusion**: Windows editors may prepend `\xef\xbb\xbf`. Decoding with plain `'utf-8'` leaves an invisible `U+FEFF` at position 0, breaking `line.startswith('>')` FASTA header checks. Use `'utf-8-sig'` to auto-strip it.
- **Windows line endings in sequences**: a trailing `\r` in a DNA line is invisible in most editors but breaks sequence length, regex matches, and `split()` results. Normalize with `.replace('\r\n', '\n').replace('\r', '\n')` before parsing.
- **Non-breaking space vs. regular space**: ` ` looks identical to `' '` but is a different byte sequence -- `split()` and exact-match database lookups treat them differently.
- **Coordinate systems**: BED is 0-based half-open; VCF/GFF is 1-based inclusive -- mixing these causes off-by-one errors independent of encoding.

## See Also

- `python-bio-file-operations` -- general file reading/writing patterns this builds on.
- `python-bio-sequences` -- working with `Seq` objects once FASTA text is decoded and clean.
- `python-bio-strings` -- string methods (`.strip()`, `.split()`, `.encode()`/`.decode()`) used throughout.
- `bio-core-biological-databases` -- building NCBI/UniProt query URLs with percent-encoded gene names.