Token Saver Memory Dedup

Token Saver Memory Dedup

Deduplicates memory records against existing memory files to prevent duplicate entries.

Purpose

This skill provides the deduplicateAgainstMemory function that filters out memory records that already exist in the memory system:

• Compares incoming records against patterns.json and gotchas.json
• Case-insensitive matching for deduplication
• Returns statistics about total, kept, and filtered records
• Handles corrupt or missing memory files gracefully

Usage

const { deduplicateAgainstMemory } = require('./scripts/main.cjs');

const records = {
  patterns: [{ text: 'Use canonical transitions', timestamp: '2026-01-01', source: 'test' }],
  gotchas: [{ text: 'Avoid shell: true for spawn', timestamp: '2026-01-01', source: 'test' }],
  issues: [],
  decisions: [],
};

const { dedupedRecords, stats } = deduplicateAgainstMemory(records, '/path/to/memory');

When to Use

• Before persisting memory records to avoid duplicates
• As part of the token-saver-context-compression workflow
• When syncing memory records between sessions

Implementation

Re-exports the deduplicateAgainstMemory function from token-saver-context-compression for module resolution and testing purposes.