Resolve Duplicates

Resolving Duplicate Entries

Use this skill when duplicate entries are detected during validation, or when you suspect multiple entries exist for the same word.

What Counts as a Duplicate?

Duplicates are entries with the same reading AND same headword. The validation script (build/validate.py) checks for this:

key = (reading, headword)  # Duplicates share both values

NOT duplicates (these are valid separate entries):

• Same reading, different headword (homophones): {橋|はし} vs {箸|はし} vs {端|はし}
• Same headword written differently: {行く|いく} vs {行く|ゆく} (different readings)
• Related but distinct words: {見る|みる} vs {見える|みえる}

Step 1: Identify Duplicates

Run validation to find duplicates:

python3 build/validate.py 2>&1 | grep -i "duplicate"

Or search manually:

# Find entries with the same reading
grep -r '"reading": "たべる"' entries/

# Find entries with similar headwords
grep -r '食べる' entries/

Step 2: Compare the Duplicate Entries

Read both (or all) entries carefully. Check:

1. Are they truly the same lexical item?

   • Same part of speech?
   • Same core meaning?
   • If they cover different senses of a polysemous word, they should be ONE entry with multiple definitions, not separate entries.

2. If they're NOT the same (rare):

   • Different parts of speech (noun vs verb homographs)
   • Genuinely different words that happen to share writing
   • In this case, differentiate by adjusting headwords or adding disambiguating notes

Step 3: Select Which Entry to Keep

Choose the entry with better quality. Evaluate:

Tie-breaker: Keep the entry with the lower ID number (older entry).

Step 4: Merge Content (If Needed)

Before deleting, check if the inferior entry has unique content worth preserving:

• Unique examples not in the better entry
• Additional usage notes or collocations
• Cross-references to other entries
• Different vocabulary tier assignment (choose the more accurate one)

If so, edit the keeper entry first to incorporate the valuable content:

# Read both entries
cat entries/00000/00123_taberu.json
cat entries/04500/04567_taberu.json

# Edit the keeper to add any missing valuable content
# Then delete the duplicate

Step 5: Delete the Duplicate

Use the delete-entry skill for safe deletion. The process:

1. Delete the entry file:

   rm entries/path/to/duplicate_entry.json
   

2. Update indexes:

   python3 build/update_indexes.py
   

3. Rebuild the flat file:

   python3 build/build_flat.py
   

4. Verify deletion:

   python3 build/validate.py
   

Complete Workflow Example

# 1. Find duplicates
python3 build/validate.py 2>&1 | grep "Duplicate"

# 2. Read both entries (example)
cat entries/00000/00123_taberu.json
cat entries/04500/04567_taberu.json

# 3. Decide: Keep 00123_taberu (better examples), delete 04567_taberu

# 4. Check if 04567_taberu has content worth merging
# (If yes, edit 00123_taberu first to add the content)

# 5. Delete the duplicate
rm entries/04500/04567_taberu.json

# 6. Update indexes and rebuild
python3 build/update_indexes.py
python3 build/build_flat.py

# 7. Validate
python3 build/validate.py

Handling Cross-References to Deleted Entries

If other entries reference the deleted duplicate:

1. Search for references:

   grep -r '"reading": "たべる"' entries/ --include="*.json" | grep cross_references
   

2. Update references to point to the kept entry's details (or remove if no longer relevant)

3. The delete-entry skill provides detailed guidance on this.

Prevention

To avoid creating duplicates in the future:

1. Always search before creating:

   grep -r '"reading": "newword"' entries/
   grep -r 'headword_kanji' entries/
   

2. Check candidate_words.json - words there may already have entries

3. Run validation frequently during entry creation sessions

Checklist

• Duplicates identified (same reading AND headword)
• Confirmed entries are for the same lexical item
• Selected which entry to keep (better quality)
• Merged any valuable unique content to keeper
• Deleted duplicate entry file
• Updated any cross-references pointing to deleted entry
• Ran update_indexes.py
• Ran build_flat.py
• Validation passes with no duplicate errors