Workflow Acquisition Archive

Workflow — Acquisition + Archive

What: Acquire from any source on planet earth, verify the bytes, preserve the metadata, normalize to an archival container, push to cold storage.

Skills used

media-ytdlp, ffmpeg-capture, decklink-tools, decklink-docs, gphoto2-tether, ndi-tools, ndi-docs, ffmpeg-streaming, mediamtx-server, ptz-visca, ptz-onvif, ffmpeg-probe, media-mediainfo, media-exiftool, ffmpeg-metadata, ffmpeg-bitstream, media-mkvtoolnix, media-gpac, media-ffmpeg-normalize, ffmpeg-transcode, media-batch, media-cloud-upload, ffmpeg-ocio-colorpro, vfx-oiio, media-ocr-ai, media-tag.

Pipeline

Step 1 — Acquire

Pick per source:

Step 2 — Verify integrity

• Full-decode pass — ffmpeg -v error -i <file> -f null - (non-zero exit = corrupt).
• Deep probe — ffmpeg-probe + media-mediainfo.
• SHA-256 checksum — store alongside the file.

Step 3 — Preserve metadata

media-exiftool — EXIF from photos, MOV udta, XMP sidecars for custom archival fields. ffmpeg-metadata for MKV global + chapter tags.

Step 4 — Normalize to archival container

MKV is the default — supports every codec, attachments, chapters, unlimited tracks. Attach the probe + mediainfo + checksum as sidecar files inside the MKV if using media-mkvtoolnix.

Step 5 — Optional codec normalization

Step 6 — Batch

media-batch (GNU parallel) fans out ingest + integrity + metadata + normalize across N sources.

Step 7 — Cold storage

media-cloud-upload:

Variants

• Continuous SDI with segmentation — 1-hour segments via segment muxer, auto-delete after 30 days.
• Automated content tagging — extract keyframes, media-tag CLIP/SigLIP per frame, consolidate tags in sidecar JSON.
• OCR verification — 0.2 fps extract, media-ocr-ai per frame, consolidate burn-in text.
• Color-space-aware archival — preserve original color tags; DO NOT auto-convert to sRGB.
• Incremental sync — rsync + post-sync checksum verification.

Gotchas

• ffmpeg -v error -i <file> -f null - is cheap integrity. Non-zero exit OR stderr with "Error" = corrupt.
• Checksums are only meaningful AT CAPTURE TIME. Checksumming an already-corrupt file is useless.
• YouTube / Vimeo re-encode on upload. --format best is NOT original quality. For true originals, you need the uploader's source.
• yt-dlp filenames can contain non-ASCII characters. Use -o "%(id)s.%(ext)s" for filesystem safety.
• --live-from-start needs source support. Most HLS windows are 20–30 min DVR — older content gone.
• DeckLink format codes are 4-letter (Hp50, Hi59, 2k24). Wrong code = silent black capture. Verify with -list_formats 1.
• 10-bit DeckLink capture requires -pix_fmt yuv422p10le AND a 10-bit-capable device.
• gphoto2 locks USB. One app claims the camera. Close EOS Utility / Lightroom / Capture One first.
• gphoto2 live-view framerate is camera-dependent (Canon ~24–30 fps, Nikon ~30 fps).
• NDI runtime (NewTek/Vizrt) is a separate install from the SDK.
• -c copy preserves codec metadata (HDR SEI, captions) but CONTAINER metadata (chapters, tags) may be lost when changing container.
• MP4 udta varies by writer (Apple / Sony / FFmpeg all different). ExifTool is the most robust reader.
• MKV global / per-track / chapter / attachment tags are all separate. Use mkvpropedit for in-place edits — ffmpeg is limited.
• ExifTool -overwrite_original deletes the backup. Keep backups until verified.
• ExifTool needs -charset filename=UTF8 for non-ASCII paths. Fails silently otherwise.
• MOV timecode lives in a tmcd track. ffmpeg -map 0 preserves; narrow stream selection drops it.
• FFV1 lossless is NOT supported by consumer players (great for LOC-grade archive, bad for distribution).
• MKV large attachments (GB-scale) slow re-muxing. For huge manifests, use separate sidecar files.
• Fragmented MP4 (via GPAC MP4Box -frag) is streamable but legacy tools may reject. Archive BOTH.
• J2K encoding is CPU-heavy (~1–2 fps/core at 4K).
• Variable-bitrate archival is storage-expensive. 60-min UHD FFV1 ≈ 500+ GB. Plan accordingly.
• AWS Deep Archive min retention 180 days. Delete before → full charge.
• Archive.org is PUBLIC. Good for public domain; do NOT upload private material.
• Multi-region replication doubles storage cost — worth it for irreplaceable originals.

Example — Archive 24-hour SDI broadcast feed

decklink-tools capture → ffmpeg segmenter → 1-hour MP4 segments → per-segment: full-decode check + SHA-256 + MediaInfo JSON sidecar → batch-upload to Glacier Deep Archive via media-cloud-upload with lifecycle rule for 30-day local purge after upload confirmation.

Related

• workflow-broadcast-delivery — the next step if the archive is to be mastered.
• workflow-live-production — the live feed that typically drives an acquisition pipeline.
• workflow-analysis-quality — integrity / spec verification.