Backup

Back up data to Dropbox

Disaster-recovery copies of expensive-to-regenerate artifacts (scraped corpora, LLM caches, geoblocked raw PDFs, BigQuery exports). Dropbox is a cold backup, not a working store. For ad-hoc file sharing / browsing, use the dropbox skill instead.

Writable remotes exist only on the host (bi-dropbox:), not the sandbox (*-ro: only). Stage compressed archives on /projects (has TBs free), not /tmp.

The two decisions: FORMAT and LOCATION

FORMAT — keyed on file count/size, not one-size-fits-all

The dividing line is Dropbox's too_many_write_operations failure (bursts of many small files). Pick by artifact shape:

Rule of thumb: per-file is strictly better until files get too many/too tiny (~a few thousand medium files is the comfortable ceiling), then you must combine. Already-compressed files (.zip, .pdf) at low count → upload per-file as-is (don't re-compress). ZSTD-9 is the sweet spot; level 19 is ~27× slower for a few %.

LOCATION — ownership/consumption, NOT "produced by a pipeline"

Fetching ≠ owning. Decide by who reads it:

• bi-dropbox:data/<x>/ — the shared data lake: raw inputs read across many projects (TSE, cnpj, court corpus) and deliberate cross-cutting namespaces (data/TCEs/ for all TCE state pulls, data/mides/ for the shared procurement bulk). These stay in data/ even if one pipeline scraped them.
• bi-dropbox:pipelines/<slug>/ — artifacts owned and consumed by only that pipeline: its build/ outputs and its own scrape/LLM caches nothing else reads. Build outputs → pipelines/<slug>/build/...; caches → pipelines/<slug>/ root.

Do NOT blanket-move data/ → pipelines/. Most of what a pipeline "produces" into data/ is shared raw. Match the existing sibling's layout (e.g. new TCE state pull → mirror the other data/TCEs/tce_<state>/ dirs).

Workflow

1. Assess shape. du -sh, find <dir> -type f | wc -l, extension breakdown, and for parquet check the codec (pq.ParquetFile(f).metadata...compression). Choose FORMAT + LOCATION from the tables above.
2. Check for prior backups first. List the destination and siblings — a previous session may already have offloaded it (possibly under a different name/layout). Reconcile before duplicating; surface (don't silently delete) anything you didn't create that overlaps.
3. Compress if needed.
   • Corpus → recompress to ZSTD parquet with the bundled helper: python recompress_zstd.py --src <clean> --dst <clean_zstd> (serial, resumable; see its header for the OOM/deadlock gotchas). Verify row counts match a sample.
   • Cache/PDF → tar -C <parent> -cf - <dir> | zstd -<L> -T3 -o <name>_<startYr>-<endYr>.tar.zst then zstd -t to verify. Name archives by time coverage.
4. Upload. rclone copy -P --transfers 4 --checkers 8 <local> <remote> (copy never deletes on remote; reserve sync for the user, with --dry-run first).
5. Verify. rclone check <local> <remote> (0 differences) or compare rclone size file counts + bytes. Small byte deltas = dir-inode accounting, not content.
6. Beware mid-build races. If the pipeline is still writing, files added after your enumeration are missed — re-scan and rclone copy the delta (it's cheap).
7. Update the data catalog ($ROOT/data_catalog/, hand-curated — edit _annotations/datasets/<x>.yaml storage.backup and the matching DATA_CATALOG.md section; do NOT regenerate). Validate: .venv/bin/python scripts/validate_annotations.py.
8. Record in memory what was non-obvious (new location, a gotcha, an audit result).

Auditing for stranded local-only data

To find artifacts that were never offloaded: reconcile local scrape/build dirs against the remote. Compare by archive/coverage, NOT raw file count — a dir of 44k loose zips correctly backed up as 2 consolidated tars will falsely look like a 44k→2 "gap". Where remote count > local, that's usually extra older pulls, not a gap. Confirm true gaps (local > remote in coverage) before uploading.

Restore

• Per-file parquet: rclone copy <remote-dir> <local> (or rclone cat one file).
• Archive: rclone cat <archive> | tar -x{--zstd or z}f - -C <parent>.

Gotchas

• too_many_write_operations fires on bursts of many files — combine into few archives.
• Foreground sleep may be blocked in-session; use background jobs / Monitor to wait.
• rclone move leaves empty dir markers — clean with rclone rmdirs.
• Commit catalog + skill changes (standing authorization); confirm before any purge/force.