Dedupe Accumulated Notes
Repeated facts are not redundancy, they are drift: the copies disagree, and a reader gets whichever greps first. Deduping means merging to one entry that keeps the best content.
Procedure
- Normalise before comparing: lowercase, strip punctuation and timestamps into a scratch copy.
- Find near-duplicates by shared key:
grep -rhoE '^[a-z][a-z0-9_-]+ *[:=]' notes/ | sort | uniq -c | sort -rn | head -20. - For long prose, cluster by a shingle hash rather than exact match, so paraphrases group too.
- For each cluster, pick the canonical entry: the one with a date, a source, and the most complete statement.
- Merge the useful bits from the others into the canonical entry; do not concatenate, edit.
- Delete the losers only after the merge is written and re-read.
- Keep a tombstone line if the key is commonly searched:
deprecated: see notes/facts/prod-db.md. - Re-run the duplicate finder to confirm the cluster is down to one.
- Fix the source that keeps re-creating the duplicate, or the cluster returns next week.
- Log the merge: key, count before, count after.
Pitfalls
Exact-match dedupe that misses three paraphrases of the same fact.
Merging by concatenating both texts, so the entry now says two different things.
Deleting the loser before writing the canonical version, losing the only dated copy.
Deduping on raw text including dates, so a fact re-confirmed each month looks unique each time.
Cleaning the output while the input process keeps emitting duplicates, so the pile regrows.
Deduping notes but not the index built from them, so search still returns the loser.
Merging entries from different scopes (prod vs staging) because their text is identical.
Rewriting duplicates to one file but leaving originals a reader still opens from git history.
Verification
grep -rhoE '^[a-z][a-z0-9_-]+ *[:=]' notes/ | sort | uniq -d | wc -l
# passes when the duplicate-key count is 0
Report to the user: the keys deduped, the before/after counts, and the source fixed to stop the regrowth.