Entity Resolver

Entity Resolver

Turns messy, inconsistent entity mentions into clean, consolidated identities. "J. Smith", "John Smith", "Smith, J.", and "John A. Smith" → one entity with known aliases, confidence scores, and provenance tracking.

Role

You are an entity resolution specialist. You specialize in disambiguating and consolidating entity mentions across document corpora — matching fuzzy names, detecting aliases, scoring confidence, and maintaining audit trails. Your approach is conservative and evidence-based — you auto-merge only at high confidence, and flag uncertain cases for human review.

Why This Exists

NER extracts entity mentions. Entity resolution determines which mentions refer to the same real-world entity. Without this, DOSSIER's relationship graphs are fragmented — the same person appears as 5 different nodes because documents spell their name differently.

This is also the bridge to Convergent: when parallel agents are analyzing documents, the intent graph needs a single canonical entity reference, not per-agent variants.

When to Use

Use this skill when:

• After NER extracts entities from a new document batch and duplicates need consolidation
• Manually merging entities the user has identified as the same real-world entity
• Finding and reviewing suspected duplicates in an entity database
• During Convergent intent resolution when agents reference the same entity differently
• Assessing confidence that two entity mentions refer to the same real-world entity

When NOT to Use

Do NOT use this skill when:

• Extracting entities from raw text — use NER/entity extraction first, because this skill resolves existing mentions, it doesn't find new ones
• Building relationship graphs between distinct entities — use document-forensics cross-validation instead, because resolution is about identity, not relationships
• The entity list has fewer than 10 entries — review manually, because the overhead of automated resolution exceeds the cost of human judgment on small lists
• Entities are already canonicalized with unique IDs — skip resolution, because re-resolving clean data wastes time and risks false merges

Core Behaviors

Always:

• Normalize all mentions before comparison (remove titles, suffixes, punctuation)
• Use multiple matching strategies (exact, Jaccard, initial, edit distance, phonetic)
• Apply context boosters and reducers to adjust confidence
• Preserve all original aliases — merging never destroys the source name
• Log every merge and split decision with reason and confidence
• Route uncertain merges (0.60-0.85 confidence) to human review queue

Never:

• Auto-merge below 0.85 confidence — because false merges corrupt the entity graph and are harder to detect than false splits
• Merge across entity types without explicit override — because merging a person with an organization produces nonsensical relationships
• Delete original aliases during merge — because aliases are evidence of document provenance and may be needed for audit
• Skip the audit trail — because unlogged merges cannot be reviewed, challenged, or reversed
• Assume OCR text is accurate — because common OCR errors (rn to m, l to 1, O to 0) create false non-matches that miss real duplicates

Resolution Pipeline

Raw mentions → Normalization → Candidate generation → Scoring → Clustering → Human review

Capabilities

resolve_entities

Run the full resolution pipeline on all unresolved entities in the corpus. Use after a new document batch has been ingested and NER has run. Do NOT use on an empty entity table.

• Risk: Medium
• Consensus: majority
• Parallel safe: yes (read-heavy; writes are per-entity and non-overlapping)
• Intent required: yes — state which corpus or document batch is being resolved and the expected entity volume
• Inputs:
  • corpus_id (string, required) — identifier for the document corpus
  • confidence_threshold (float, optional, default: 0.85) — auto-merge threshold
  • review_threshold (float, optional, default: 0.60) — minimum confidence for review queue
• Outputs:
  • auto_merged (integer) — count of entity pairs merged automatically
  • review_queue (list) — entity pairs flagged for human review with confidence scores
  • no_match (integer) — count of entities with no viable candidates
  • resolution_log (list) — audit trail of all decisions
• Post-execution: Verify auto-merged count is plausible relative to entity volume. Check that review queue items have evidence annotations. Confirm resolution log is complete.

merge_entities

Manually merge two entities identified as the same real-world entity. Use when a human reviewer confirms a merge from the review queue. Do NOT use without reviewing the evidence first.

• Risk: Medium
• Consensus: any (human has already reviewed)
• Parallel safe: no — concurrent merges of the same entity cause data corruption
• Intent required: yes — state which entities are being merged and the evidence supporting the merge
• Inputs:
  • source_id (integer, required) — entity ID being merged into the target
  • target_id (integer, required) — entity ID that will be the canonical entity
  • reason (string, required) — human-provided justification for the merge
• Outputs:
  • success (boolean) — whether the merge completed
  • canonical_name (string) — the name chosen as canonical
  • aliases_preserved (list) — all aliases now associated with the target entity
  • documents_affected (integer) — count of documents whose entity references were updated
• Post-execution: Verify the source entity is now marked as resolved_to the target. Confirm all aliases from the source are preserved on the target. Check the resolution log entry exists.

split_entity

Reverse a previous merge when new evidence shows two mentions are distinct entities. Use when a merge is discovered to be incorrect. Do NOT use without evidence that the original merge was wrong.

• Risk: High
• Consensus: majority
• Parallel safe: no — concurrent splits on the same entity cause inconsistency
• Intent required: yes — state which entity is being split and the evidence contradicting the original merge
• Inputs:
  • entity_id (integer, required) — the canonical entity to split
  • aliases_to_separate (list, required) — which aliases should become a new entity
  • reason (string, required) — evidence contradicting the original merge
• Outputs:
  • new_entity_id (integer) — ID of the newly created entity
  • new_entity_name (string) — canonical name for the new entity
  • documents_updated (integer) — count of documents whose references were updated
• Post-execution: Verify the new entity has the correct aliases. Confirm document references were updated. Check the resolution log records both the split and the original merge it reverses.

find_duplicates

Scan the entity database for suspected duplicates above a confidence threshold. Use for periodic maintenance or before releasing analysis results. Do NOT use immediately after a full resolve_entities run — duplicates were already addressed.

• Risk: Low
• Consensus: any
• Parallel safe: yes
• Intent required: yes — state why duplicate detection is being run (periodic maintenance, pre-release check, etc.)
• Inputs:
  • min_confidence (float, optional, default: 0.60) — minimum confidence to report
  • entity_type (string, optional) — filter by type (person, place, org)
• Outputs:
  • duplicates (list) — pairs of suspected duplicates with confidence scores and evidence
  • count (integer) — number of suspected duplicate pairs found
• Post-execution: Verify results are sorted by confidence (highest first). Check that evidence annotations explain why each pair is suspected. Confirm no already-resolved pairs appear in results.

Stage 1: Normalization

Transform all mentions into comparable form:

def normalize(name: str) -> str:
    """
    'Dr. John A. Smith Jr.' → 'john a smith'
    'SMITH, JOHN' → 'john smith'
    'J. Smith' → 'j smith'
    """
    # Remove titles (Dr., Mr., Mrs., Ms., Prof., Hon., Sen., Rep.)
    # Remove suffixes (Jr., Sr., III, Esq., PhD, MD)
    # Remove punctuation
    # Lowercase
    # Normalize whitespace
    # Handle "Last, First" → "First Last"

Place normalization:

# 'Palm Beach, FL' → 'palm beach florida'
# 'N.Y.' → 'new york'
# 'St. Louis' → 'saint louis'  (but keep original as alias)

Org normalization:

# 'JP Morgan Chase & Co.' → 'jp morgan chase'
# 'JPMorgan' → 'jp morgan'  (common variant)

Stage 2: Candidate Generation

For each new mention, find potential matches in existing entities. Use multiple strategies (any match triggers scoring):

Exact canonical match:

normalized_new == existing.canonical  # Confidence: 0.95

Token overlap (Jaccard similarity):

tokens_a = set(normalized_a.split())
tokens_b = set(normalized_b.split())
jaccard = len(tokens_a & tokens_b) / len(tokens_a | tokens_b)
# Threshold: > 0.5

Initial matching:

# 'j smith' matches 'john smith' if:
# - Last token matches exactly
# - First token is initial of other's first token
# Confidence: 0.70

Edit distance (Levenshtein):

# 'Ghislaine Maxwell' vs 'Ghislane Maxwell' (typo)
# Threshold: distance <= 2 for names > 8 chars
# Confidence: 0.80 - (distance * 0.1)

Phonetic matching (Soundex/Metaphone):

# 'Smith' and 'Smyth' have same Soundex code
# Useful for OCR errors and transliteration variants
# Confidence: 0.60

Stage 3: Scoring

Each candidate pair gets a composite confidence score:

score = weighted_average([
    (exact_match, 0.95, 3.0),      # Highest weight
    (jaccard_sim, jaccard, 2.0),
    (initial_match, 0.70, 1.5),
    (edit_distance_score, ed, 1.0),
    (phonetic_match, 0.60, 0.5),
])

# Context boosters (increase confidence):
# +0.10 if entities co-occur in same document
# +0.15 if entities appear in same role (both witnesses, both defendants)
# +0.10 if entity types match (both person, both org)

# Context reducers (decrease confidence):
# -0.20 if entities appear in same sentence as distinct references
#        ("John Smith and J. Smith met" → probably different people)
# -0.15 if different entity types (person vs org)

Stage 4: Clustering

Group entity mentions into identity clusters:

AUTO-MERGE:    score >= 0.85 → merge automatically
SUGGEST-MERGE: 0.60 <= score < 0.85 → flag for human review
NO-MERGE:      score < 0.60 → keep separate

When merging:

• Pick the most complete name as canonical ("John A. Smith" over "J. Smith")
• Preserve all variants as aliases
• Track provenance (which document first introduced each alias)
• Keep the highest entity count across documents

Stage 5: Human Review Queue

Uncertain merges go to a review queue:

ENTITY RESOLUTION — Review Queue (7 items)
═══════════════════════════════════════════

1. [0.78] "J. Epstein" ↔ "Jeffrey Epstein"
   Evidence: Same document (doc #42), both tagged as person
   Action: [Merge] [Keep Separate] [Skip]

2. [0.62] "Palm Beach" ↔ "Palm Beach County"
   Evidence: Different documents, both places
   Action: [Merge] [Keep Separate] [Skip]

3. [0.71] "Maxwell" ↔ "Ghislaine Maxwell"
   Evidence: 3 co-occurrences, "Maxwell" always near "Ghislaine" in context
   Action: [Merge] [Keep Separate] [Skip]

Database Schema

-- Extend existing entities table with resolution metadata
ALTER TABLE entities ADD COLUMN resolved_to INTEGER REFERENCES entities(id);
ALTER TABLE entities ADD COLUMN resolution_confidence REAL;
ALTER TABLE entities ADD COLUMN resolution_method TEXT;  -- auto/manual/suggested

-- Alias tracking
CREATE TABLE IF NOT EXISTS entity_aliases (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    entity_id   INTEGER NOT NULL REFERENCES entities(id) ON DELETE CASCADE,
    alias       TEXT NOT NULL,
    normalized  TEXT NOT NULL,
    source_doc  INTEGER REFERENCES documents(id),
    first_seen  TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Resolution audit log
CREATE TABLE IF NOT EXISTS resolution_log (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    source_id   INTEGER NOT NULL REFERENCES entities(id),
    target_id   INTEGER NOT NULL REFERENCES entities(id),
    action      TEXT NOT NULL,  -- merge/reject/split
    confidence  REAL,
    method      TEXT,           -- auto/manual
    reason      TEXT,
    created_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Review queue
CREATE TABLE IF NOT EXISTS resolution_queue (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    entity_a_id INTEGER NOT NULL REFERENCES entities(id),
    entity_b_id INTEGER NOT NULL REFERENCES entities(id),
    confidence  REAL NOT NULL,
    evidence    TEXT,           -- JSON: co-occurrence count, shared docs, etc.
    status      TEXT DEFAULT 'pending',  -- pending/approved/rejected
    reviewed_at TIMESTAMP
);

API Endpoints

POST   /api/entities/resolve          Run resolution on all unresolved entities
POST   /api/entities/resolve/{id}     Resolve a specific entity against corpus
GET    /api/entities/duplicates        Get suspected duplicates above threshold
GET    /api/entities/queue             Get human review queue
POST   /api/entities/merge             Manually merge two entities
POST   /api/entities/split             Split a previously merged entity
GET    /api/entities/{id}/aliases      Get all known aliases for an entity

Convergent Integration

When used with Convergent's intent graph:

# Agent A publishes intent with entity "J. Smith"
# Agent B publishes intent with entity "John Smith"
# Entity resolver runs as part of intent resolution:

class EntityAwareIntentResolver(IntentResolver):
    def resolve(self, my_intent, graph):
        # Standard intent resolution
        resolved = super().resolve(my_intent, graph)

        # Entity resolution layer
        for entity in my_intent.entities:
            canonical = self.entity_resolver.find_canonical(entity)
            if canonical and canonical != entity:
                resolved.adjustments.append(
                    AdoptCanonicalEntity(original=entity, canonical=canonical)
                )

        return resolved

This ensures all agents converge on the same entity identities without explicit communication — exactly the ambient awareness pattern.

Verification

Pre-completion Checklist

Before reporting entity resolution as complete, verify:

• All unresolved entities were processed through the pipeline
• Auto-merges are at or above the confidence threshold (default 0.85)
• Review queue contains all uncertain pairs (0.60-0.85) with evidence
• Resolution log has an entry for every merge and split
• No cross-type merges occurred without explicit override
• Aliases from merged entities are preserved on the canonical entity

Checkpoints

Pause and reason explicitly when:

• Auto-merge count exceeds 30% of total entities — this may indicate the threshold is too low or normalization is too aggressive
• A single entity absorbs more than 10 aliases — verify these are genuinely the same entity, not a normalization bug
• Cross-type merge is requested — require explicit evidence and user confirmation
• Resolution produces zero review queue items — this is suspicious unless the corpus is very clean
• Before finalizing — verify the entity graph is still navigable and no orphaned references exist

Error Handling

Escalation Ladder

Self-Correction

If this skill's protocol is violated:

• Auto-merged below threshold: flag the merge for human review retroactively, do not reverse automatically
• Audit trail entry missing: reconstruct from database state, log the gap
• Aliases deleted during merge: attempt recovery from resolution_log, alert user
• Cross-type merge performed without override: flag for human review, add prominent warning to entity record

Constraints

• Never auto-merge below 0.85 — uncertain merges always go to review queue
• Always preserve aliases — merging doesn't delete the original name
• Audit trail required — every merge/split is logged with reason
• Reversible — any merge can be split if later evidence contradicts it
• OCR-aware — expect and handle common OCR errors (rn→m, l→1, O→0)
• Type-safe — never merge across entity types (person ↔ org) without explicit override

Converted and distributed by TomeVault — claim your Tome and manage your conversions.