CSV Cleaner
Two-phase workflow: diagnose first, clean second. Never start fixing before knowing what is broken.
Phase 1: Diagnose
Run scripts/diagnose.py on the input file:
python scripts/diagnose.py <path-to-file>
The script reports:
- Encoding (utf-8, latin-1, utf-16, etc.)
- Delimiter (comma, tab, semicolon)
- Quote character and escape behavior
- Header row detection (present, absent, or duplicate)
- Row count
- Column count consistency across rows
- Type inference per column (integer, float, date, string, mixed)
- Null distribution per column
- Duplicate row count
Read the output carefully before doing anything else. The diagnosis tells you which fixes apply.
Phase 2: Clean
Apply fixes in this order:
- Fix encoding: re-read with detected encoding, write as utf-8.
- Normalize line endings to
\n.
- Strip BOM and zero-width characters.
- Fix headers: lowercase, snake_case, replace spaces with underscores, ensure uniqueness.
- Trim whitespace from every cell.
- Coerce types where the column is type-consistent.
- Standardize null representations: empty string, "NA", "null", "n/a" all become empty.
- Deduplicate exact rows.
- Validate column count consistency. Flag rows that do not match the header.
Do not silently change data. If a fix alters values, report the count.
Output
Save the cleaned file as <original-name>-cleaned.csv in the same directory. Always utf-8, always comma-delimited, always with a header row.
Then emit a summary:
## Diagnosis
<key findings from phase 1>
## Fixes applied
- <fix>: <count of rows or cells affected>
- <fix>: <count>
## Manual review needed
- <issue that the script could not fix safely>
When to refuse
If the file appears to be a different format (Excel binary, JSON, fixed-width), do not try to clean it as CSV. Tell the user what the file actually is and suggest the right tool.
If the file has fewer than 5 rows after deduplication, ask the user whether the input is correct before continuing.
1---2name: csv-cleaner3description: Diagnose and clean messy CSV, TSV, or spreadsheet files. Use whenever the user uploads a CSV or TSV that has data quality issues, asks to "clean", "normalize", "fix", or "deduplicate" tabular data, or hands over a file with encoding problems, broken headers, type inconsistencies, duplicate rows, or missing values. Triggers on "clean this CSV", "fix this data", "this spreadsheet is messy", "deduplicate these rows", or any data-quality request on a tabular file. Always run scripts/diagnose.py before making changes so the cleanup is informed by what is actually broken.4---56# CSV Cleaner78Two-phase workflow: diagnose first, clean second. Never start fixing before knowing what is broken.910## Phase 1: Diagnose1112Run `scripts/diagnose.py` on the input file:1314```bash15python scripts/diagnose.py <path-to-file>16```1718The script reports:1920- Encoding (utf-8, latin-1, utf-16, etc.)21- Delimiter (comma, tab, semicolon)22- Quote character and escape behavior23- Header row detection (present, absent, or duplicate)24- Row count25- Column count consistency across rows26- Type inference per column (integer, float, date, string, mixed)27- Null distribution per column28- Duplicate row count2930Read the output carefully before doing anything else. The diagnosis tells you which fixes apply.3132## Phase 2: Clean3334Apply fixes in this order:35361. Fix encoding: re-read with detected encoding, write as utf-8.372. Normalize line endings to `\n`.383. Strip BOM and zero-width characters.394. Fix headers: lowercase, snake_case, replace spaces with underscores, ensure uniqueness.405. Trim whitespace from every cell.416. Coerce types where the column is type-consistent.427. Standardize null representations: empty string, "NA", "null", "n/a" all become empty.438. Deduplicate exact rows.449. Validate column count consistency. Flag rows that do not match the header.4546Do not silently change data. If a fix alters values, report the count.4748## Output4950Save the cleaned file as `<original-name>-cleaned.csv` in the same directory. Always utf-8, always comma-delimited, always with a header row.5152Then emit a summary:5354````55## Diagnosis56<key findings from phase 1>5758## Fixes applied59- <fix>: <count of rows or cells affected>60- <fix>: <count>6162## Manual review needed63- <issue that the script could not fix safely>64````6566## When to refuse6768If the file appears to be a different format (Excel binary, JSON, fixed-width), do not try to clean it as CSV. Tell the user what the file actually is and suggest the right tool.6970If the file has fewer than 5 rows after deduplication, ask the user whether the input is correct before continuing.
0xelitesystem