Data Hygiene

Data Hygiene

Clean before you enrich, and keep it clean with a queue rather than a quarterly panic.

When to use

• Before any paid enrichment run, every time
• Bounce rates climbing
• Duplicate accounts breaking account-level reporting
• Ongoing, as a scheduled loop

Inputs

• Reads: workspace/data/object-model.md and the data dictionary, both required for the freshness windows
• Needs from user: the list or export to clean, and the suppression sources

Workflow

1. The pre-paid pass, in this order

Order matters. Dedup after normalization finds more, and suppression before enrichment saves money.

1. Normalize domains        strip protocol, www, params, lowercase, resolve redirects
2. Normalize titles         map to a seniority band and a function
3. Normalize geography      to a consistent country and region scheme
4. Flag invalid emails      syntax, disposable, role-based, free-mail on B2B rows
5. Dedup accounts           on the normalized key, keep the most complete record
6. Dedup contacts           on verified email, then on name plus account
7. Apply suppression        customers, open opportunities, do-not-contact, competitors
8. Report drop reasons      every removed row carries why

Step 8 is the one people omit and the one that makes the pass auditable. A cleanup that removes 40% of rows with no reason column is indistinguishable from a bug.

2. Merge, do not delete

When deduping, merge into the most complete record. Field-level merge rules come from the direction-of-truth column in the data dictionary. Keep the losing record IDs in a crosswalk so historical activity and reporting still resolve.

Deleting duplicates orphans their activity history, and the activity history is often the most valuable thing on the record.

3. Build the decay loop, not a schedule

A quarterly full refresh is expensive and mostly wasted. Freshness-driven refresh is cheaper and better.

For each field, from the data dictionary:
  freshness window → age of current value → stale or current

Refresh queue = stale fields on records that matter
                prioritized by tier, then by staleness
                capped per run

Fields decay at very different rates. A person's job title decays fast, a company's industry barely decays at all, and refreshing them on the same cadence means paying for one and under-serving the other.

Reference decay rates:

4. Watch for silent divergence

Run a weekly distribution check on the fields that drive routing and scoring. If a field's value distribution shifts more than 20% week over week, something upstream changed, and the dangerous version of that is a sync writing a default value successfully.

Monitoring for errors will not catch this. Monitor the shape of the data.

5. Report the four numbers

Rows in / rows out, with drop reasons
Duplicate rate before and after
Invalid email rate
Records outside their freshness window

Track them over time. A hygiene pass that reports only this run is a chore. Tracked over quarters it tells you whether the intake is getting cleaner or whether you are mopping under a running tap.

Output

• Writes: workspace/data/hygiene-report.md, cleaned list, workspace/data/merge-crosswalk.csv
• Uses: templates/data-quality-scorecard.csv
• Prints: the four numbers, the drop-reason breakdown, the refresh queue size, and any distribution anomalies

Rules & quality bar

• Runs before any paid enrichment. Every time, no exceptions
• Normalize before dedup. Pre-normalization duplicate counts are always wrong
• Merge, never delete. Keep the crosswalk
• Every dropped row carries a reason
• Refresh is a staleness queue, not a scheduled full rerun
• Watch distributions, not just errors

Related skills

• Requires: data-model-and-sync
• Runs before: enrichment-waterfall
• See also: docs/metrics.md, docs/cost-model.md