Data Cleanup and Formatting
Standardize messy tabular data without ever silently changing what it says. Every transformation is declared; every ambiguous value is surfaced, not guessed.
Steps
- Get the data (file path or paste) and profile it first: row count, columns, detected types, distinct-value oddities (mixed date formats, case-inconsistent names, stray whitespace, mixed types in one column), missing-value counts, and candidate duplicates. Present this profile BEFORE changing anything.
- Propose the cleanup plan as a checklist the user confirms: target date format, name casing, type per column, duplicate rule (exact vs. fuzzy key), and missing-value policy per column (leave blank / fill with sentinel / drop row — never a silent default).
- Apply the confirmed plan. For anything ambiguous (is
02/03/24Feb 3 or Mar 2? are "J. Smith" and "John Smith" the same person?), stop and ask — wrong-but-tidy is worse than messy. - Deliver: the cleaned dataset, plus a transformation log — rows in/out, per-column changes applied, duplicates found (listed, not just deleted), missing values and how each was handled, and any rows quarantined as unparseable rather than mangled.
- Verify: spot-check that no value changed meaning (dates shifted, names merged wrongly); re-state the row count arithmetic (in = out + dropped + quarantined) so nothing vanishes.
Constraints
- Never delete or merge rows without listing exactly which ones and why.
- Never guess an ambiguous date, unit, or identity — ask.
- The original input is never overwritten; cleaned output is a new file/table.
- If the data is too large to show fully, show the profile + a sample and operate via a script the user can inspect, not invisible edits.
Full walkthrough, examples, and variations: recipes/Recipe-009-Data-Cleanup-Formatting.md.