Scientific file formats in code
File formats are decade-scale decisions: data outlives the code that
wrote it, and a format choice made in a script today determines
whether a dataset is readable, FAIR and efficient in ten years. The
guiding principle is SELF-DESCRIBING data: a file a stranger can open
and understand - variables named, units attached, provenance noted -
without emailing the author (rseng-data-management owns the
surrounding practice; this skill owns the format engineering).
Choosing a format
- Multidimensional arrays (grids, time series stacks, images,
simulation output): NetCDF (atmosphere/ocean/climate lingua
franca, built on HDF5) or HDF5 directly; both hierarchical,
self-describing, partial-read capable and language-portable.
- Tabular data at scale: Parquet - columnar, typed, compressed,
schema-carrying; the upgrade path from CSV when files grow or
types matter.
- Small human-facing tables and interchange: CSV is fine - WITH a
stated dialect (delimiter, encoding, quoting) and a data
dictionary alongside (rseng-data-management).
- Domain standards first: if the field has one (NeXus for photon/
neutron science, CF-governed NetCDF in climate, community formats
generally), emitting it beats inventing anything - it is what
colleagues' tools already read (rseng-fair-software's I).
Formats to migrate away from when encountered: pickles as storage
(unreadable outside Python, version-fragile, unsafe to load from
strangers), unversioned homegrown binaries, MAT files as long-term
archives, spreadsheets as databases. Flag them, explain the failure
mode, offer the migration.
Metadata: the self-describing part
- Attach units, long names and fill values to every variable at
write time - in code, not in a README written later. CF
conventions define exactly how for NetCDF and are checkable with
automated compliance checkers; run one in CI when a project's
outputs claim CF compliance (rseng-ci-cd).
- Record provenance in file attributes: producing software and
version, input identifiers, creation time, configuration
hash - the file should testify about its own origin.
- Keep schemas versioned: when a project's file layout evolves, add
a format-version attribute, and keep readers for old versions or
a migration script (rseng-legacy-code discipline applied to data).
Performance engineering: chunking and compression
- Chunk to match access patterns: time-slice reads want chunks along
time; map reads want spatial chunks. Wrong chunking makes reads
orders of magnitude slower on large stores - decide from how the
data will be READ, not written.
- Compression is usually free performance for scientific data
(gzip/zstd-class codecs); test level trade-offs on real data, and
prefer bit-shuffle-style filters for floats where available.
- For cloud or parallel access, chunked stores (HDF5/NetCDF-4 and
their cloud-optimized descendants in the Pangeo ecosystem) enable
partial and concurrent reads - the pattern behind scalable
analysis (rseng-big-data-processing).
Testing format code
Round-trip tests (write, read back, compare with tolerances -
rseng-numerical-accuracy), a checked-in small golden file to catch
accidental format changes, and reading files produced by OTHER tools
in the ecosystem as compatibility tests (rseng-testing).
Working with this skill
This skill is source-independent: its authority is the format
specifications and community conventions linked below.
Learn more (verified):
Related skills
Check whether any of these applies before moving on:
- rseng-big-data-processing - chunked stores enable scalable reads
- rseng-data-management - surrounding dataset practice and deposit
- rseng-fair-software - domain standards serve interoperability
- rseng-legacy-code - schema versioning and old-format readers
- rseng-numerical-accuracy - round-trip tests need float tolerances
- rseng-testing - golden-file and compatibility tests
1---2name: rseng-scientific-file-formats3description: Covers choosing and handling scientific data formats in code: HDF5 and NetCDF for array data, CF conventions and standard metadata, Parquet for tabular data, domain standards (NeXus and similar), self-describing files, chunking and compression choices, and migrating away from fragile formats like pickles and ad-hoc binaries. Use when the user chooses a file format for research data, reads or writes HDF5/NetCDF/Parquet/zarr-style stores, asks about chunking, compression or metadata embedding, or ships data in CSV, pickle, MAT or homegrown binary formats that deserve scrutiny. (The surrounding data practice - versioning, deposit, licensing, documentation - is rseng-data-management.)4license: CC-BY-4.05---67# Scientific file formats in code89File formats are decade-scale decisions: data outlives the code that10wrote it, and a format choice made in a script today determines11whether a dataset is readable, FAIR and efficient in ten years. The12guiding principle is SELF-DESCRIBING data: a file a stranger can open13and understand - variables named, units attached, provenance noted -14without emailing the author (rseng-data-management owns the15surrounding practice; this skill owns the format engineering).1617## Choosing a format1819- Multidimensional arrays (grids, time series stacks, images,20 simulation output): NetCDF (atmosphere/ocean/climate lingua21 franca, built on HDF5) or HDF5 directly; both hierarchical,22 self-describing, partial-read capable and language-portable.23- Tabular data at scale: Parquet - columnar, typed, compressed,24 schema-carrying; the upgrade path from CSV when files grow or25 types matter.26- Small human-facing tables and interchange: CSV is fine - WITH a27 stated dialect (delimiter, encoding, quoting) and a data28 dictionary alongside (rseng-data-management).29- Domain standards first: if the field has one (NeXus for photon/30 neutron science, CF-governed NetCDF in climate, community formats31 generally), emitting it beats inventing anything - it is what32 colleagues' tools already read (rseng-fair-software's I).3334Formats to migrate away from when encountered: pickles as storage35(unreadable outside Python, version-fragile, unsafe to load from36strangers), unversioned homegrown binaries, MAT files as long-term37archives, spreadsheets as databases. Flag them, explain the failure38mode, offer the migration.3940## Metadata: the self-describing part4142- Attach units, long names and fill values to every variable at43 write time - in code, not in a README written later. CF44 conventions define exactly how for NetCDF and are checkable with45 automated compliance checkers; run one in CI when a project's46 outputs claim CF compliance (rseng-ci-cd).47- Record provenance in file attributes: producing software and48 version, input identifiers, creation time, configuration49 hash - the file should testify about its own origin.50- Keep schemas versioned: when a project's file layout evolves, add51 a format-version attribute, and keep readers for old versions or52 a migration script (rseng-legacy-code discipline applied to data).5354## Performance engineering: chunking and compression5556- Chunk to match access patterns: time-slice reads want chunks along57 time; map reads want spatial chunks. Wrong chunking makes reads58 orders of magnitude slower on large stores - decide from how the59 data will be READ, not written.60- Compression is usually free performance for scientific data61 (gzip/zstd-class codecs); test level trade-offs on real data, and62 prefer bit-shuffle-style filters for floats where available.63- For cloud or parallel access, chunked stores (HDF5/NetCDF-4 and64 their cloud-optimized descendants in the Pangeo ecosystem) enable65 partial and concurrent reads - the pattern behind scalable66 analysis (rseng-big-data-processing).6768## Testing format code6970Round-trip tests (write, read back, compare with tolerances -71rseng-numerical-accuracy), a checked-in small golden file to catch72accidental format changes, and reading files produced by OTHER tools73in the ecosystem as compatibility tests (rseng-testing).7475## Working with this skill7677This skill is source-independent: its authority is the format78specifications and community conventions linked below.7980Learn more (verified):81 - https://cfconventions.org - CF metadata conventions82 - https://www.unidata.ucar.edu/software/netcdf/ - NetCDF83 - https://www.hdfgroup.org/solutions/hdf5/ - HDF584 - https://parquet.apache.org - Apache Parquet85 - https://www.nexusformat.org - NeXus domain format86 - https://pangeo.io - Pangeo community (cloud-optimized scientific87 data practice)8889<!-- related-skills:begin -->9091## Related skills9293Check whether any of these applies before moving on:9495- rseng-big-data-processing - chunked stores enable scalable reads96- rseng-data-management - surrounding dataset practice and deposit97- rseng-fair-software - domain standards serve interoperability98- rseng-legacy-code - schema versioning and old-format readers99- rseng-numerical-accuracy - round-trip tests need float tolerances100- rseng-testing - golden-file and compatibility tests101102<!-- related-skills:end -->
fdiblen