agentskillexchange Profile and clean large CSV datasets from the terminal with qsv
Inspect, profile, normalize, and diff large CSV files before loading them into downstream analytics or automation workflows.
Prerequisites
qsv binary and CSV datasets
Installation
Use the upstream install or setup path that matches your environment:
- cargo install zipsign
- brew install qsv
- conda install conda-forge::qsv
- git clone https://github.com/dathere/qsv.git
Requirements and caveats from upstream:
- | py✨📇🔣 | Create a new computed column or filter rows by evaluating a Python expression on every row of a CSV file. Python's [f-strings](https://www.freecodecamp.org/news/python-f-strings-tutori...
- 🔣: requires UTF-8 encoded input.
Basic usage or getting-started notes:
| describegpt📇🗃️🤖🌐🪄📚⛩️ | Infer a "neuro-symbolic" Data Dictionary, De...
| luau✨📇🌐🔣📚 | Create multiple new computed columns, filter rows, compute aggregations and build complex...
| moarstats📇🏎️ | Add up to an additional 55 statistical measures, including extended outlier, robust & bivariate statistics to an existing stats CSV file. ([example](docs/moarstats/NYC_31...
Source: https://github.com/dathere/qsv
Extracted from upstream docs: https://raw.githubusercontent.com/dathere/qsv/HEAD/README.md