LarrySequino CSV Profiler
Answers one question: can anything be built on this extract, and where will it break.
Procedure
- Count rows and columns before reading anything else, and compare against what the sender claimed. A silent truncation at export time is common and invisible later.
- Infer a type per column from a sample, then verify it against the whole column. The column that is an integer in the first thousand rows and a string in row 40,000 is the one that breaks the load.
- Report null rate per column. A column that is 98 percent null is not a column.
- Check the claimed key for duplicates and nulls. If it has either, say so before anyone writes a join against it.
- Range-check numerics and dates. Dates in the future and negative quantities are the two that survive every other check.
- Flag free-text columns. They carry delimiters, newlines, and personal data, and each of those is a separate problem.
See the delimiter and encoding notes for the cases where a file parses cleanly and is still wrong.
Output
A table with one row per column: type, null rate, distinct count, min, max, and a note where the column failed a check. Then a short verdict: usable, usable with caveats, or send it back.