Duplicate Ad Creative Finder

Duplicate Ad Creative Finder

Collapses a list of platform ads down to the distinct creatives actually behind them.

Meta and LinkedIn mint a fresh ad ID every time the same image is pointed at a new campaign, audience or placement. Reporting on ad IDs therefore tells you which placements ran, never which creative worked. On a real B2B library, 2,824 creative links resolved to 1,006 unique ads and only 202 distinct creatives — a 5× reduction. One image was running under 17 different ad IDs across both platforms.

Run this before tagging and before any creative analysis. Everything downstream depends on it.

1 · Operating contract

• Group by pixels, never by metadata. Filenames, ad names and campaign names are not identity. The same creative is routinely renamed; different creatives are routinely named the same.
• Never merge without producing the audit. Every merge decision must be checkable by a human looking at two thumbnails side by side. A silent merge is an unfalsifiable claim.
• Never partially update the outputs. The four tables are generated together and replaced together. Asset IDs are sequential; a partial update silently desynchronises them.
• Never estimate a hash. Fingerprinting requires actual computation. Where the runtime cannot execute code, emit the procedure and the file list and say the step was not run.
• Preserve every original ad ID. Deduplication reduces the unit of analysis, never the data. Every ad ID must remain traceable to its creative.

2 · When to run this

Run at the start of any creative analysis project, whenever new creative is added to a library, and before ad-creative-tagger — always before, never after.

Do not re-run it when new performance data arrives for creative already fingerprinted. Identity is a property of the creative, not the reporting period.

3 · Required inputs and preflight

Must have

Improves the output

Ad name, campaign, placement, launch date, and the caption text — which becomes copy_id and makes copy-vs-visual-impact-analyzer possible later.

Preflight

1. URLs resolvable. Sample 20 links and confirm they fetch. Halt if a systematic pattern fails — see §5 on case-mangled IDs, which silently kills half a dataset.
2. Carousel completeness. Confirm every carousel has all its cards, in order. Halt on partial decks.
3. Media types identified. Confirm each row is classified image, video or carousel before hashing; the three take different treatments.
4. Storage available. Confirm a cache location exists. Without one the run is not repeatable and every re-run re-downloads everything.

4 · The procedure

Step 1 — Repair the source links

Before downloading anything, check for systematically broken IDs.

The canonical failure: an export arrives with lowercased file IDs. Drive IDs are case-sensitive, so every one of them 404s, and if you are not watching, half the dataset silently drops out. Rebuild the true casing by scraping the folder listing and matching on the lowercased form.

Report how many links were repaired. A large number means the export path needs fixing upstream.

Step 2 — Download and cache

Fetch every file in parallel — 16 threads is a reasonable default — into a content cache keyed by a hash of the URL. Convert share links to direct-download endpoints, and handle virus-scan confirmation tokens where the host uses them.

Nothing is ever downloaded twice. This is what makes the run repeatable in seconds rather than hours.

Step 3 — Fingerprint the pixels

Cache the fingerprints separately from the media. Re-running the grouping logic should then cost seconds, not a re-download.

Step 4 — Group by visual identity

• Images: group where Hamming distance ≤ 6 on a 64-bit hash. Tighten to ≤ 4 if the library contains many near-identical variants that must stay distinct.
• Video: require all three keyframe hashes to match, and durations to agree within a small tolerance. One frame collides too easily — title cards and fades are near-identical across unrelated ads.
• Carousels: match each card first, then treat the carousel's identity as the ordered sequence of its card IDs. Two carousels match only if every card matches in the same order. A reordered deck is a different creative.

Step 5 — Mint stable IDs

Give each surviving creative a stable, human-readable ID: IMG-1001, VID-1001, CAR-1001, with carousel cards as CAR-1001-1 … -N pointing at their parent.

These IDs are permanent. Once assigned, an ID is never reused or renumbered, because it will end up in spreadsheets, briefs and other people's decks.

Step 6 — Pick the surviving URL

A creative usually has several URLs pointing at it. Choose one canonical file, preferring the copy you control over a CDN copy that may expire. Record which URLs were merged into it.

Step 7 — Extract copy identity

Where captions are supplied, deduplicate them the same way — normalize whitespace, casing and punctuation, then hash — and assign copy_id. This produces the four-level model:

asset_id     one unique visual                IMG-/VID-/CAR-####
copy_id      one unique caption               COPY-####
creative_id  one unique (asset + copy) pair   CRV-####
ad_id        one platform ad

The relationship is genuinely many-to-many, and keeping all four levels is what makes matched-pair analysis possible later.

Step 8 — Build the audit

Produce a row per original ad showing the assigned creative's thumbnail beside that ad's own original file. Any row where the two images differ is a bad match.

To audit fast: filter to merged duplicates only. Those are the only rows where a match can be wrong — representative files are true by definition. On a 1,006-ad run that is 806 rows to check, not 1,006.

5 · Computation rules

Non-negotiable.

• Fan results back out. Tag and analyze at creative level, then map results back to every ad ID. Never discard ad IDs.
• Hash the pixels, not the file bytes. Byte hashes differ for re-encodes of the same image and are useless here.
• Never group across media types. A video and its thumbnail are not the same creative.
• Case sensitivity is not a detail. File IDs from most cloud hosts are case-sensitive; any pipeline that lowercases them is silently destroying rows.
• Regenerate all four outputs together, always.

6 · Output contract

Four tables, generated together.

Plus a run summary:

DEDUPLICATION · Jan–Aug 2026 · Meta + LinkedIn
  source rows     2,824
  unique ads      1,006
  unique creatives  202     → 5.0× reduction
  links repaired    506     (case-mangled Drive IDs)
  most reused       IMG-1043 · 17 ads across both platforms
  audit rows to check  806  (merged duplicates only)

7 · Failure and degraded states

8 · Worked example

Input: 2,824 rows exported from Meta and LinkedIn, each with a creative link.

Step 1 finds 506 Meta rows carrying lowercased Drive file IDs. All 506 would have 404'd, dropping the entire Meta half of the dataset. Casing restored by matching against the folder listing.

Step 3–4 fingerprints 2,824 files. Images group at Hamming ≤ 6. One carousel pair matches on five of six cards — left separate, because a false merge is unrecoverable.

Step 5 yields 202 creatives: 148 images, 31 videos, 23 carousels.

Step 8 produces 806 audit rows. Spot-checking finds two bad matches, both near-identical seasonal variants of one template. The threshold tightens to ≤ 4 and the run is repeated — from cached fingerprints, in seconds.

Result: reporting moves from 1,006 ad rows to 202 creative rows. Tagging cost drops by 80%, and every creative now carries one consistent set of tags instead of a dozen slightly different ones.

9 · Self-check

1. Were links repaired before download, and is the repair count reported?
2. Was grouping done on perceptual hashes, never on names or byte hashes?
3. Do carousels match on the ordered sequence of every card?
4. Is every original ad ID still traceable to its creative?
5. Were all four tables regenerated together?
6. Does the audit table show the assigned thumbnail beside each ad's own original?
7. Is the Hamming threshold recorded in the run summary?

MIT licensed.