How do I install the Lit Dedup skill?

Run npx skillmds@latest add kennethkhoocy/lit-dedup in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Lit Dedup safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Lit Dedup?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Lit Dedup free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Lit Dedup?

kennethkhoocy (@kennethkhoocy) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Lit Dedup by kennethkhoocy — Agent Skill

Name: Lit Dedup
Author: kennethkhoocy

Lit Dedup

Stage 5 of the lit review pipeline: merge and deduplicate papers from all prior stages (Undermind, Scholar Labs, supplementary search) into a single master list with provenance tracking. Two-pass dedup: exact DOI match then LLM fuzzy match via DeepSeek/Claude. Only use this skill when explicitly requested — e.g., the user says "run lit-dedup", "lit-dedup", or "/lit-dedup". Do NOT auto-trigger on general literature review requests.

kennethkhoocy Updated Sep 17, 2026

File contents

Lit-Dedup (Stage 5 — Merge & Deduplicate)

Merge multiple pipeline stage outputs into one deduplicated master list with provenance tracking.

Input: one or more JSON files from Stages 1–4 Output: merged_results.json + merged_results.ris + dedup_log.json

Quick Start

# Merge all JSON files in a directory
python scripts/lit_dedup.py --input-dir ./results/ -o merged_results.json

# Merge specific files
python scripts/lit_dedup.py --inputs stage1.json stage2.json stage4.json -o merged.json

# DOI-only dedup (skip LLM pass)
python scripts/lit_dedup.py --inputs *.json --no-llm -o merged.json

# Non-interactive (skip confirmation)
python scripts/lit_dedup.py --inputs *.json -o merged.json --yes

Prerequisites

pip install aiohttp

API Keys

DEEPSEEK_API_KEY — primary (DeepSeek chat API for fuzzy matching)
ANTHROPIC_API_KEY — fallback (Claude Sonnet via Anthropic API)

CLI Flags

Flag	Default	Description
`--inputs`	—	Input JSON files (nargs='+')
`--input-dir`	—	Directory to glob *.json from
`-o, --output`	`merged_results.json`	Output JSON path
`--model`	`deepseek-chat`	LLM model override
`--concurrency`	`100`	Max simultaneous LLM requests
`--no-llm`	off	Skip Pass 2 (DOI-only dedup)
`--yes`	off	Skip HITL confirmation checkpoint
`--emit-pairs PATH`	—	Agent-driven: write candidate pairs to PATH and stop before the LLM pass (no API)
`--ingest-verdicts EMIT VERDICTS`	—	Agent-driven: merge the subagent pair verdicts and write all outputs (no API)

Two-Pass Dedup

Pass 1 — DOI match: normalize DOIs, group by exact match, merge using priority (published > forthcoming > working paper)
Pass 2 — LLM fuzzy match: pre-filter candidate pairs by author surname overlap or title Jaccard > 0.4, then judge each pair for a same-paper verdict

Model routing (agent-driven default)

In the orchestrator's agent-driven flow the Pass-2 same-paper judgment is made by a subagent, not the API — a Sonnet subagent by default, promoted to Opus when the GUI's Use Opus for all tasks (all_opus) is set. Run --emit-pairs to dump the candidate pairs, judge each {i, j, a, b} pair with the subagent (yes/no duplicate), then --ingest-verdicts to run the union-find merge and write every output. The DeepSeek/Sonnet API path is the autonomous fallback for standalone runs. See the orchestrator SKILL.md ("How it runs" → "Model routing").

Output Schema

Same as pipeline stages, with added fields:

{
  "title": "Paper Title",
  "authors": "Author1, Author2",
  "year": "2024",
  "doi": "10.1234/...",
  "abstract": "...",
  "journal": "Journal Name",
  "url": "https://...",
  "sources": ["undermind", "scholarlabs", "ssrn"],
  "alt_url": "https://ssrn.com/..."
}

Output Files

merged_results.json — deduplicated master list
merged_results.ris — RIS for Zotero import
dedup_log.json — log of all merge decisions and stats

---name: lit-dedupdescription: Stage 5 of the lit review pipeline: merge and deduplicate papers from all prior stages (Undermind, Scholar Labs, supplementary search) into a single master list with provenance tracking. Two-pass dedup: exact DOI match then LLM fuzzy match via DeepSeek/Claude. Only use this skill when explicitly requested — e.g., the user says "run lit-dedup", "lit-dedup", or "/lit-dedup". Do NOT auto-trigger on general literature review requests.---# Lit-Dedup (Stage 5 — Merge & Deduplicate)Merge multiple pipeline stage outputs into one deduplicated master listwith provenance tracking.**Input**: one or more JSON files from Stages 1–4**Output**: `merged_results.json` + `merged_results.ris` + `dedup_log.json`## Quick Start```bash# Merge all JSON files in a directorypython scripts/lit_dedup.py --input-dir ./results/ -o merged_results.json# Merge specific filespython scripts/lit_dedup.py --inputs stage1.json stage2.json stage4.json -o merged.json# DOI-only dedup (skip LLM pass)python scripts/lit_dedup.py --inputs *.json --no-llm -o merged.json# Non-interactive (skip confirmation)python scripts/lit_dedup.py --inputs *.json -o merged.json --yes```## Prerequisites```bashpip install aiohttp```## API Keys- `DEEPSEEK_API_KEY` — primary (DeepSeek chat API for fuzzy matching)- `ANTHROPIC_API_KEY` — fallback (Claude Sonnet via Anthropic API)## CLI Flags| Flag | Default | Description ||------|---------|-------------|| `--inputs` | — | Input JSON files (nargs='+') || `--input-dir` | — | Directory to glob *.json from || `-o, --output` | `merged_results.json` | Output JSON path || `--model` | `deepseek-chat` | LLM model override || `--concurrency` | `100` | Max simultaneous LLM requests || `--no-llm` | off | Skip Pass 2 (DOI-only dedup) || `--yes` | off | Skip HITL confirmation checkpoint || `--emit-pairs PATH` | — | Agent-driven: write candidate pairs to PATH and stop before the LLM pass (no API) || `--ingest-verdicts EMIT VERDICTS` | — | Agent-driven: merge the subagent pair verdicts and write all outputs (no API) |## Two-Pass Dedup1. **Pass 1 — DOI match**: normalize DOIs, group by exact match, merge using priority (published > forthcoming > working paper)2. **Pass 2 — LLM fuzzy match**: pre-filter candidate pairs by author surname overlap or title Jaccard > 0.4, then judge each pair for a same-paper verdict## Model routing (agent-driven default)In the orchestrator's agent-driven flow the Pass-2 same-paper judgment is made by a**subagent**, not the API — a **Sonnet subagent by default**, promoted to **Opus**when the GUI's *Use Opus for all tasks* (`all_opus`) is set. Run `--emit-pairs` todump the candidate pairs, judge each `{i, j, a, b}` pair with the subagent (yes/noduplicate), then `--ingest-verdicts` to run the union-find merge and write everyoutput. The DeepSeek/Sonnet API path is the autonomous fallback for standalone runs.See the orchestrator SKILL.md ("How it runs" → "Model routing").## Output SchemaSame as pipeline stages, with added fields:```json{ "title": "Paper Title", "authors": "Author1, Author2", "year": "2024", "doi": "10.1234/...", "abstract": "...", "journal": "Journal Name", "url": "https://...", "sources": ["undermind", "scholarlabs", "ssrn"], "alt_url": "https://ssrn.com/..."}```## Output Files- `merged_results.json` — deduplicated master list- `merged_results.ris` — RIS for Zotero import- `dedup_log.json` — log of all merge decisions and stats

Frequently asked questions