Checksum Data Across the Pipeline
Bytes corrupt in transit, on disk, and during transformation; only a checksum re-computed on the far side detects it. Verify at each boundary so a corrupt byte fails the hop that introduced it instead of surfacing during a restore.
Procedure
- Choose the algorithm by purpose: SHA-256 for integrity and provenance, xxHash/CRC32C for high-throughput transit, MD5 only as a legacy fallback.
- Compute at the source before upload and store the digest alongside metadata:
sha256sum data.csv > data.csv.sha256. - Verify after each transfer:
sha256sum -c data.csv.sha256on the destination, oraws s3api head-object --checksum-mode ENABLEDto read the storedChecksumSHA256. - Use the transport's own integrity where present — S3 multipart uses per-part checksums,
rsync -cre-hashes, TLS protects transit but not at-rest. - Force end-to-end verification on object upload:
aws s3 cp --checksum-algorithm SHA256, and preferaws s3api put-objectwithChecksumSHA256so the service validates. - Re-verify at restore time from the archive's manifest, not from a fresh hash, so a corrupt archive is caught rather than trusted.
- Store digests in a manifest written once at creation, and treat any mismatch as a hard failure that halts the pipeline.
- For derived data, checksum inputs and the transform version together so the same output is reproducible.
Pitfalls
- Trusting TLS for at-rest integrity; TLS protects the wire, and a bad disk block reads back wrong after a clean transfer.
- Using the object store's
ETagas if it were always the file MD5 — multipart uploads make it a composite, not a plain hash. - Computing the checksum after decompression on one side and before on the other, so the digests never match by construction.
- Storing the hash in the same object/row as the data, so corruption that hits the data usually hits the hash too.
- Verifying at upload and never at read, so a bit flip that happens months later goes undetected until restore.
- A weak hash (CRC32) used where deliberate tampering matters; CRC32 is for errors, not adversaries.
- Dropping the manifest during a migration, losing the reference that made verification possible.
Verification
sha256sum -c data.csv.sha256 # expect: data.csv: OK
aws s3api head-object --bucket acme --key data.csv \
--checksum-mode ENABLED --query 'ChecksumSHA256'
Source and destination digests match, the manifest stores the original values, and a deliberately corrupted sample fails -c with FAILED.
Report: "Verified objects across hops; all checksums match source manifest; injected corruption on was caught at ."