# Dedupe

> Deduplication reference — exact matching, fuzzy matching, hash-based dedup, bloom filters, and data quality. Use when removing duplicate records, files, or data entries.

- Skill: `bytesagain/dedupe` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add bytesagain/dedupe`
- Raw SKILL.md: https://api.skillmd.com/api/skills/bytesagain/dedupe/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: bytesagain (https://skillmd.com/u/bytesagain)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/bytesagain/dedupe

---


# Dedupe — Data Deduplication Reference

Quick-reference skill for deduplication strategies, algorithms, and data quality patterns.

## When to Use

- Removing duplicate rows from datasets or databases
- Deduplicating files in storage systems
- Implementing fuzzy matching for near-duplicate detection
- Choosing between exact and probabilistic dedup methods
- Building ETL pipelines with deduplication stages

## Commands

### `intro`

```bash
scripts/script.sh intro
```

Overview of deduplication — types, strategies, and tradeoffs.

### `exact`

```bash
scripts/script.sh exact
```

Exact deduplication — hash-based, key-based, and sorting approaches.

### `fuzzy`

```bash
scripts/script.sh fuzzy
```

Fuzzy deduplication — similarity measures, blocking, and record linkage.

### `files`

```bash
scripts/script.sh files
```

File-level deduplication — fdupes, jdupes, rdfind, and storage dedup.

### `algorithms`

```bash
scripts/script.sh algorithms
```

Dedup algorithms — bloom filters, HyperLogLog, MinHash, SimHash.

### `sql`

```bash
scripts/script.sh sql
```

SQL deduplication patterns — ROW_NUMBER, DISTINCT, GROUP BY strategies.

### `cli`

```bash
scripts/script.sh cli
```

Command-line dedup tools — sort, uniq, awk, and stream processing.

### `checklist`

```bash
scripts/script.sh checklist
```

Deduplication quality checklist and validation steps.

### `help`

```bash
scripts/script.sh help
```

### `version`

```bash
scripts/script.sh version
```

## Configuration

| Variable | Description |
|----------|-------------|
| `DEDUPE_DIR` | Data directory (default: ~/.dedupe/) |

---

*Powered by BytesAgain | bytesagain.com | hello@bytesagain.com*