Record Dataset Provenance
A number you cannot trace to one specific, hashable snapshot of one specific dataset is not evidence — it is a memory. Pin the source before you compute.
Procedure
- For each input, add a
PROVENANCE.mdrow: dataset name, publisher, exact retrieval URL (endpoint, not homepage), UTC retrieval time, license, and version/revision id. - Hash the bytes and keep the manifest:
shasum -a 256 data/*.csv | tee data/CHECKSUMS.txt. If upstream edits the file, the hash must change — a stable hash is your guarantee you analysed a frozen copy. - Capture the query, not just the URL. Store the full request so it is replayable:
curl -sG 'https://api.example/v2/rows' --data-urlencode 'since=2024-01-01' -o raw.json - Record every transform. Each derived table gets one line: source file, filter, row count in → row count out. Unexplained row shrinkage is the most common silent bug.
- State the license and its limits (
CC-BY-4.0requires attribution;CC-BY-NC-4.0forbids commercial use; "research use only" forbids redistribution). - Note known collection issues from the publisher's own doc: coverage gaps, imputation, sampling design, revision policy.
- Freeze: copy raw inputs into
data/raw/and make downstream code read only from there.
mkdir -p data/raw && cp /tmp/download/*.csv data/raw/
shasum -a 256 data/raw/*.csv > data/RAW_MANIFEST.sha256
wc -l data/raw/*.csv # record baseline counts next to the manifest
Pitfalls
- Re-pulling "the same" endpoint silently returns a newer revision; without a hash you cannot tell the dataset changed under you.
- Vendored rows from a second-hand mirror (Kaggle, Hugging Face) may be an undeclared subsample — always reach the original.
- Licenses attach to a specific version; a later release can relicence.
- A CSV whose row count differs from the stated
nmeans dropped/filtered rows you have not accounted for. - Timestamps without a timezone are ambiguous; store UTC and say so.
Verification
shasum -a 256 -c data/RAW_MANIFEST.sha256
All checksums OK and PROVENANCE.md has a license + version for every file. Report: "Analysis on dataset v, sha256 , retrieved , licence ; 11,204 raw rows."