Detect a Stale Knowledge Base
An index answers from the corpus as it was at build time, not as it is now. Staleness is invisible until an answer contradicts a document, so it must be measured on a schedule.
Procedure
Store the source hash and mtime per document at ingest, e.g.
sha256sumof the raw file in the index record.On each check, recompute the source hashes:
find docs -type f -name '*.md' -exec sha256sum {} + | sort > /tmp/now.sha.Diff against the index's recorded hashes to classify each document as
unchanged,changed,deleted, ornew.Treat
changedanddeletedas correctness bugs: the index still serves retired text as current.Age the drift: if the newest index build is older than the freshness SLO (e.g. 24h for a live KB), alert regardless of diffs.
Report the ratio drifted/total; above a threshold (say 2%) the whole index needs a rebuild, not a patch.
Feed the
changedset to the incremental refresh path;deletedmust remove the chunk ids, not just orphan them.Watch the answer side too: a chunk that cites a source revision that no longer exists is a stale citation.
Schedule the check where the drift cannot be silently skipped — a cron with an alert on non-zero drift.
Record every check to
kb/freshness.jsonlso staleness has a history, not just a current value.Alert on the age of the newest successful build, not only on a diff, so a stopped ingester is caught.
Pitfalls
Hashing normalised text instead of the raw file, so a formatting-only change looks unchanged while content moved.
Rebuilding on mtime alone; a copied file gets a fresh mtime with stale contents.
Removing a document from the index but leaving its chunk ids resolvable, so dead citations still click.
Checking freshness only when someone complains, which is after the wrong answer shipped.
Trusting a source's own "updated" field rather than the bytes you actually ingested.
A rebuild that succeeds but writes to a new collection while serving still reads the old one.
A freshness check that runs but whose exit code is ignored by the scheduler.
Treating a document moved between directories as unchanged because the content hash matched.
Verification
python3 freshness.py --index index/ --docs docs/ --slo-hours 24
# passes when drift ratio is 0 for unchanged sources and every changed/deleted doc is queued for refresh
Report to the user: documents scanned, drift ratio, the list of changed and deleted docs, and the age of the last build.