lamm-mit Minerals Data — Structured CSV Querying
Query and analyze structured CSV datasets from the critical minerals corpus. Supports listing available datasets, describing schemas, filtering, grouping, and aggregation via pandas.
Usage
List available datasets:
python3 {baseDir}/scripts/query_data.py --list
Describe a dataset:
python3 {baseDir}/scripts/query_data.py --dataset usgs/production.csv --describe
Query with DSL:
python3 {baseDir}/scripts/query_data.py --dataset usgs/production.csv --query "groupby:commodity|agg:value:sum|sort:value:desc|head:10"
Filter with pandas expression:
python3 {baseDir}/scripts/query_data.py --dataset usgs/production.csv --filter "year >= 2022"
Combine filter and query:
python3 {baseDir}/scripts/query_data.py --dataset usgs/trade.csv --filter "commodity == 'lithium'" --query "groupby:country|agg:value:sum|sort:value:desc|head:5"
Parameters
| Parameter | Description | Default |
|---|---|---|
--list |
List all available CSV datasets | - |
--dataset |
Path to CSV file (relative to corpus dir) | - |
--describe |
Show schema, dtypes, sample rows, statistics | - |
--query |
Pipe-delimited DSL for pandas operations | - |
--filter |
Pandas query expression for filtering | - |
--corpus-dir |
Directory containing data files | ~/critical-minerals-data/ |
--format |
Output format: table, json, csv | table |
Query DSL
Pipe-delimited operations that map to pandas:
| Operation | Syntax | Example |
|---|---|---|
| Group by | groupby:col |
groupby:commodity |
| Aggregate | agg:col:func |
agg:value:sum |
| Sort | sort:col:dir |
sort:value:desc |
| Head | head:n |
head:10 |
| Select columns | select:col1,col2 |
select:commodity,value |
Functions: sum, mean, count, min, max, median, std
Examples
# Top producing countries for lithium
python3 {baseDir}/scripts/query_data.py --dataset usgs/production.csv \
--filter "commodity == 'lithium'" \
--query "groupby:country|agg:value:sum|sort:value:desc|head:10"
# Year-over-year trade data
python3 {baseDir}/scripts/query_data.py --dataset comtrade/exports.csv \
--query "groupby:year|agg:value:sum|sort:year:asc"
# Dataset overview
python3 {baseDir}/scripts/query_data.py --dataset worldbank/indicators.csv --describe
Notes
- Requires
pandas>=2.0.0(already in ScienceClaw requirements) - CSV catalog is cached at
~/critical-minerals-data/.csv_catalog.json - Handles encoding fallbacks: UTF-8, Latin-1, CP1252
- Filter expressions are sanitized to prevent code injection