Refresh an Index Incrementally
Full rebuilds are correct but expensive; incremental refresh is the same result for the diff. The danger is orphans — chunks that belong to a document that changed or vanished but were never removed.
Procedure
Compare source hashes to the index (
detect-a-stale-knowledge-base) to get thechanged,deletedandnewsets.For each changed document, delete all existing chunk ids with that
source_uribefore inserting the new chunks.Upsert by the stable chunk id (
attach-stable-citation-ids) so an unchanged chunk is overwritten, not duplicated.Embed only the new chunk texts; reuse cached embeddings for identical content (
cache-embeddings-by-content-hash).Apply deletes first, then inserts, in one batch per document so a crash cannot leave both old and new versions live.
Commit the index and the source-hash manifest together; if the manifest advances but the index does not, the next run will skip the change.
Rebuild only on schema changes — chunker, embedding model, or metadata shape — not on ordinary edits.
Verify post-refresh that the index holds no chunk whose
source_urino longer resolves to a file.Run the retrieval eval after a refresh; a broken delete shows up as duplicate or stale hits.
Keep a rebuild path that works from scratch, so a corrupted index can be discarded, not repaired.
Record the refresh as a transaction id per document so a partially applied batch can be retried exactly.
Pitfalls
Inserting new chunks without deleting the old ones, so the index quietly doubles and returns both versions.
Advancing the hash manifest before the index commit, so a crash loses the change permanently.
Re-embedding every chunk because the loader cannot tell which text changed.
Deleting by source path while the path was renamed, leaving the old path's chunks alive forever.
Running the refresh in the same collection without a version tag, so a partial failure is undetectable.
Assuming upsert is delete-then-insert when the backend only inserts, creating hidden duplicates.
Retrying a refresh that failed after the deletes but before the inserts, losing content until the retry lands.
Advancing the source manifest when the embedding call fails, so the changed document is marked done.
Verification
python3 refresh.py --index index/ --manifest kb/hashes.json
ls index/orphans.txt 2>/dev/null && echo "ORPHANS PRESENT" || echo "no orphans"
# passes when orphan count is 0 and eval recall is unchanged from the pre-refresh baseline
Report to the user: documents changed, chunks deleted and inserted, embeddings computed vs reused, and the eval delta.