How do I install the Dedupe skill?

Run npx skillmds@latest add bytesagain/dedupe in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Dedupe safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: executes scripts. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Dedupe?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Dedupe free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Dedupe?

bytesagain (@bytesagain) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Dedupe by bytesagain — Agent Skill

Name: Dedupe
Rating: 4.5 (4 reviews)
Author: bytesagain

File contents

Dedupe — Data Deduplication Reference

Quick-reference skill for deduplication strategies, algorithms, and data quality patterns.

When to Use

Removing duplicate rows from datasets or databases
Deduplicating files in storage systems
Implementing fuzzy matching for near-duplicate detection
Choosing between exact and probabilistic dedup methods
Building ETL pipelines with deduplication stages

Commands

`intro`

scripts/script.sh intro

Overview of deduplication — types, strategies, and tradeoffs.

`exact`

scripts/script.sh exact

Exact deduplication — hash-based, key-based, and sorting approaches.

`fuzzy`

scripts/script.sh fuzzy

Fuzzy deduplication — similarity measures, blocking, and record linkage.

`files`

scripts/script.sh files

File-level deduplication — fdupes, jdupes, rdfind, and storage dedup.

`algorithms`

scripts/script.sh algorithms

Dedup algorithms — bloom filters, HyperLogLog, MinHash, SimHash.

`sql`

scripts/script.sh sql

SQL deduplication patterns — ROW_NUMBER, DISTINCT, GROUP BY strategies.

`cli`

scripts/script.sh cli

Command-line dedup tools — sort, uniq, awk, and stream processing.

`checklist`

scripts/script.sh checklist

Deduplication quality checklist and validation steps.

`help`

scripts/script.sh help

`version`

scripts/script.sh version

Configuration

Variable	Description
`DEDUPE_DIR`	Data directory (default: ~/.dedupe/)

Powered by BytesAgain | bytesagain.com | hello@bytesagain.com

---name: dedupedescription: Deduplication reference — exact matching, fuzzy matching, hash-based dedup, bloom filters, and data quality. Use when removing duplicate records, files, or data entries.--- # Dedupe — Data Deduplication Reference Quick-reference skill for deduplication strategies, algorithms, and data quality patterns. ## When to Use - Removing duplicate rows from datasets or databases - Deduplicating files in storage systems - Implementing fuzzy matching for near-duplicate detection - Choosing between exact and probabilistic dedup methods - Building ETL pipelines with deduplication stages ## Commands ### `intro` ```bash scripts/script.sh intro ``` Overview of deduplication — types, strategies, and tradeoffs. ### `exact` ```bash scripts/script.sh exact ``` Exact deduplication — hash-based, key-based, and sorting approaches. ### `fuzzy` ```bash scripts/script.sh fuzzy ``` Fuzzy deduplication — similarity measures, blocking, and record linkage. ### `files` ```bash scripts/script.sh files ``` File-level deduplication — fdupes, jdupes, rdfind, and storage dedup. ### `algorithms` ```bash scripts/script.sh algorithms ``` Dedup algorithms — bloom filters, HyperLogLog, MinHash, SimHash. ### `sql` ```bash scripts/script.sh sql ``` SQL deduplication patterns — ROW_NUMBER, DISTINCT, GROUP BY strategies. ### `cli` ```bash scripts/script.sh cli ``` Command-line dedup tools — sort, uniq, awk, and stream processing. ### `checklist` ```bash scripts/script.sh checklist ``` Deduplication quality checklist and validation steps. ### `help` ```bash scripts/script.sh help ``` ### `version` ```bash scripts/script.sh version ``` ## Configuration | Variable | Description | |----------|-------------| | `DEDUPE_DIR` | Data directory (default: ~/.dedupe/) | --- *Powered by BytesAgain | bytesagain.com | hello@bytesagain.com*

Frequently asked questions

Dedupe

File contents

Dedupe — Data Deduplication Reference

When to Use

Commands

intro

exact

fuzzy

files

algorithms

sql

cli

checklist

help

version