Data Formats
How to work with diverse and unknown data formats.
Format Detection
Always inspect before parsing:
file <filename> # MIME type detection
xxd <filename> | head -5 # hex dump (first bytes)
head -3 <filename> # text preview
python3 -c "
with open('<filename>', 'rb') as f:
h = f.read(16)
print(h, h.hex())
"
Common Formats
Binary
- Magic bytes: Most binary formats start with a signature (ELF:
\x7fELF, PNG: \x89PNG)
- Endianness: Check if little-endian or big-endian (
struct.unpack('<I', ...) vs '>I')
- Alignment: Fields are often aligned to 4 or 8 bytes
- Offsets: Binary headers often contain offsets to other sections
Structured text
- CSV/TSV: Check delimiter (comma, tab, pipe), quoting, header row
- JSON:
python3 -c "import json; json.load(open('f'))"
- YAML: Check indentation, anchors/aliases
- TOML:
python3 -c "import tomllib; ..."
- XML: Check encoding declaration, namespaces
Checkpoints / Model files
- PyTorch:
.pt, .pth → torch.load(f, map_location='cpu')
- TensorFlow:
.ckpt → index + data files, use tf.train.load_checkpoint()
- NumPy:
.npy, .npz → numpy.load()
- HuggingFace:
config.json + model.safetensors
- ONNX:
onnx.load()
Database files
- SQLite:
file says "SQLite 3.x database" → sqlite3 <file> ".tables"
- WAL files: SQLite write-ahead log — recover with
sqlite3 PRAGMA
- CSV dumps: Often need schema inference
Parsing Strategies
Unknown binary format
- Hex dump first 256 bytes:
xxd file | head -16
- Look for magic bytes, version numbers, string tables
- Check file size — does it suggest a pattern? (e.g., N * record_size)
- Look for documentation of the format online
- Write a minimal parser, test on known values
Large structured files
- Never load entirely — sample first:
head, tail, shuf -n 10
- Check consistency: are all lines the same format?
- Count fields:
head -1 file | awk -F',' '{print NF}'
- Watch for: mixed types, missing values, encoding issues
Multi-file datasets
- List all files and sizes
- Look for manifest/index files (often JSON or CSV)
- Check naming patterns — timestamps, sequence numbers, shards
- Process one file first, then generalize
Common Pitfalls
- Assuming UTF-8 when the file is Latin-1 or binary
- Assuming CSV when it's TSV (or vice versa)
- Ignoring the header row
- Not handling quoted fields with embedded delimiters
- Reading binary files as text (corrupts data)
- Endianness mismatch (x86 is little-endian, network byte order is big-endian)
1---2name: data-formats3description: "Working with diverse data formats: binary, text, structured, and custom"4tags: [data, parsing, formats, benchmark]5version: "1.0.0"6metadata:7 source: https://skillmd.com/skills/vstorm-co/data-formats8---910# Data Formats1112How to work with diverse and unknown data formats.1314## Format Detection1516Always inspect before parsing:1718```bash19file <filename> # MIME type detection20xxd <filename> | head -5 # hex dump (first bytes)21head -3 <filename> # text preview22python3 -c "23with open('<filename>', 'rb') as f:24 h = f.read(16)25 print(h, h.hex())26"27```2829## Common Formats3031### Binary32- **Magic bytes**: Most binary formats start with a signature (ELF: `\x7fELF`, PNG: `\x89PNG`)33- **Endianness**: Check if little-endian or big-endian (`struct.unpack('<I', ...)` vs `'>I'`)34- **Alignment**: Fields are often aligned to 4 or 8 bytes35- **Offsets**: Binary headers often contain offsets to other sections3637### Structured text38- **CSV/TSV**: Check delimiter (comma, tab, pipe), quoting, header row39- **JSON**: `python3 -c "import json; json.load(open('f'))"`40- **YAML**: Check indentation, anchors/aliases41- **TOML**: `python3 -c "import tomllib; ..."`42- **XML**: Check encoding declaration, namespaces4344### Checkpoints / Model files45- **PyTorch**: `.pt`, `.pth` → `torch.load(f, map_location='cpu')`46- **TensorFlow**: `.ckpt` → index + data files, use `tf.train.load_checkpoint()`47- **NumPy**: `.npy`, `.npz` → `numpy.load()`48- **HuggingFace**: `config.json` + `model.safetensors`49- **ONNX**: `onnx.load()`5051### Database files52- **SQLite**: `file` says "SQLite 3.x database" → `sqlite3 <file> ".tables"`53- **WAL files**: SQLite write-ahead log — recover with `sqlite3` PRAGMA54- **CSV dumps**: Often need schema inference5556## Parsing Strategies5758### Unknown binary format591. Hex dump first 256 bytes: `xxd file | head -16`602. Look for magic bytes, version numbers, string tables613. Check file size — does it suggest a pattern? (e.g., N * record_size)624. Look for documentation of the format online635. Write a minimal parser, test on known values6465### Large structured files661. Never load entirely — sample first: `head`, `tail`, `shuf -n 10`672. Check consistency: are all lines the same format?683. Count fields: `head -1 file | awk -F',' '{print NF}'`694. Watch for: mixed types, missing values, encoding issues7071### Multi-file datasets721. List all files and sizes732. Look for manifest/index files (often JSON or CSV)743. Check naming patterns — timestamps, sequence numbers, shards754. Process one file first, then generalize7677## Common Pitfalls7879- Assuming UTF-8 when the file is Latin-1 or binary80- Assuming CSV when it's TSV (or vice versa)81- Ignoring the header row82- Not handling quoted fields with embedded delimiters83- Reading binary files as text (corrupts data)84- Endianness mismatch (x86 is little-endian, network byte order is big-endian)