undefined-ui Backfill an archive
A backfill is where cost and quality both go wrong quietly. Done in the wrong order it also inverts the vault's history, because later sources should update pages that earlier ones created.
Core rule
Oldest first, ten sources per batch, stop for a go-ahead between batches.
Workflow
- Triage before ingesting. Read titles and first paragraphs. Sort into worth-ingesting, keep-in-raw, and delete. Most archives are half dead links and things saved but never opened.
- Order by date, oldest first. Run it the other way and every old source arrives as a contradiction against a page that already has the final answer.
- Batch of ten. Ingest following
second-brain-ingest. - Checkpoint: update
index.md, append tolog.md, commit with the batch range in the message, report counts. - Stop and wait before the next batch.
- Full lint at the end. Large imports always leave orphans and duplicates.
Output format
Batch <n>/<total>: <range>
Ingested: <n> sources -> <n> new pages, <n> updated, <n> links
Skipped: <n> (reason)
Running total: <n> of <n>
Next batch: <what it contains>
Calibration
Read every page the first batch produced, all of them. A flaw you let through in batch one is a flaw in four hundred pages by the end, and fixing it afterwards costs more than the original run.
If the archive is over about two hundred sources, propose a filter before starting rather than quoting the full cost.