PortalJS — Check Data Quality
Overview
Run a read-only quality audit of one CSV or TSV file, local or remote, and return a
structured JSON report. The audit profiles every column — null/blank counts, inferred
value types, numeric ranges, likely year/date fields — and flags duplicate rows,
duplicate values in identifier-like columns, ambiguous overlapping year columns (e.g.
calendar year vs fiscal year), and mixed-type columns. It never edits the source
file, datasets.json, or any other project file; it only reads the target file (a
remote URL is downloaded to a temp file that is deleted before the run ends) and
prints a report. Use it before publishing a dataset with portaljs-add-dataset, or to
diagnose why a showcase renders wrong.
Prerequisites
python3 on PATH — the audit logic runs as an embedded Python script; nothing is
installed.- One CSV or TSV file, given as a local path or an
http/https URL. Only one file
per run.
Instructions
The canonical, full step-by-step workflow is
.claude/commands/portaljs-check-data-quality.md —
the single source of truth. Read and follow it when executing. Summary:
- Gather input — the file path or URL to audit. If missing, ask for it; never dead-end.
- Resolve the source: if it's an
http/https URL, download it to a temp file first;
otherwise use the local path as given.
- Validate the extension is
.csv or .tsv. If not, or the file is missing, or the
header row is empty, stop and surface the error JSON as-is — do not guess a fix.
- Profile every column: null/blank counts, distinct values, sample values, inferred
per-value type (boolean/integer/float/date/string), numeric min/max, and year
range for columns whose name looks year-like.
- Derive findings from the profiles — duplicate rows, missing-value ratios, invalid
year values, mixed types, suspect negative values, duplicate identifier values, and
ambiguous overlapping year columns — each tagged
critical, warning, or info.
- Assemble the JSON report (
status, file metadata, findings, recommendations,
column_profiles), print it, and clean up the temp file if one was created.
- Relay the report to the user as-is; do not modify the source file,
datasets.json,
or any other project file based on the findings — that's a separate, explicit step.
Output
A single JSON object printed to stdout:
status — ok, warning, or critical.file, file_name, source_type (local or url), row_count, column_count.findings — structured issues, most severe first.recommendations — de-duplicated suggested next steps.column_profiles — per-column summary (nulls, blanks, distinct count, sample
values, inferred types, numeric/year ranges).
No files are created or modified. A remote URL's temp download is removed on exit,
success or failure alike.
Error Handling
| Symptom |
Cause |
Fix |
"File ... is not available." |
Local path is wrong, or the URL download failed |
Verify the path or URL is reachable and retry. |
"Only CSV and TSV files are supported right now." |
File extension isn't .csv/.tsv |
Convert the file, or point to its tabular source instead. |
"... does not contain tabular headers." |
File is empty or the header row is malformed |
Open the file and confirm it has a valid, non-empty header line. |
| Command hangs on a URL |
Remote host is slow or blocks non-browser requests |
Download the file manually and audit the local copy instead. |
python3: command not found |
Python 3 isn't installed or not on PATH |
Install Python 3, or run the audit where it's available. |
| Report looks truncated in the terminal |
Large report wrapped/paginated by the shell |
Redirect to a file (> report.json) and open it separately. |
Examples
Example 1 — Audit a local CSV before publishing
/portaljs-check-data-quality ./public/data/trash.csv
Example 2 — Audit a remote CSV over HTTPS
/portaljs-check-data-quality https://example.com/trash.csv
Example 3 — Audit a TSV and save the report for review
bash scripts/check-data-quality.sh ./data/emissions.tsv > /tmp/emissions-quality.json
Example 4 — Read a critical status report
{
"status": "critical",
"findings": [
{ "severity": "critical", "check": "duplicate_rows", "message": "42 duplicate rows found." }
],
"recommendations": ["Review and deduplicate repeated rows if they are not intentional."]
}
Fix the flagged rows/columns, then re-run the audit before publishing.
Resources
1---2name: portaljs-check-data-quality3description: Audit a local or remote tabular file (CSV/TSV) for common data quality issues — schema, nulls, types, duplicates. Read-only. Use when a dataset needs a quality check before publishing, or a showcase renders wrong (blank cells, garbled numbers, an unsortable date column) and the cause needs isolating.4license: MIT5---6
7# PortalJS — Check Data Quality
8
9## Overview
10
11Run a read-only quality audit of one CSV or TSV file, local or remote, and return a
12structured JSON report. The audit profiles every column — null/blank counts, inferred
13value types, numeric ranges, likely year/date fields — and flags duplicate rows,
14duplicate values in identifier-like columns, ambiguous overlapping year columns (e.g.
15`calendar year` vs `fiscal year`), and mixed-type columns. It never edits the source
16file, `datasets.json`, or any other project file; it only reads the target file (a
17remote URL is downloaded to a temp file that is deleted before the run ends) and
18prints a report. Use it before publishing a dataset with `portaljs-add-dataset`, or to
19diagnose why a showcase renders wrong.
20
21## Prerequisites
22
23- `python3` on `PATH` — the audit logic runs as an embedded Python script; nothing is
24 installed.
25- One CSV or TSV file, given as a local path or an `http`/`https` URL. Only one file
26 per run.
27
28## Instructions
29
30The canonical, full step-by-step workflow is
31[`.claude/commands/portaljs-check-data-quality.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/portaljs-check-data-quality.md) —
32the single source of truth. Read and follow it when executing. Summary:
33
341. Gather input — the file path or URL to audit. If missing, ask for it; never dead-end.
352. Resolve the source: if it's an `http`/`https` URL, download it to a temp file first;
36 otherwise use the local path as given.
373. Validate the extension is `.csv` or `.tsv`. If not, or the file is missing, or the
38 header row is empty, stop and surface the error JSON as-is — do not guess a fix.
394. Profile every column: null/blank counts, distinct values, sample values, inferred
40 per-value type (boolean/integer/float/date/string), numeric min/max, and year
41 range for columns whose name looks year-like.
425. Derive findings from the profiles — duplicate rows, missing-value ratios, invalid
43 year values, mixed types, suspect negative values, duplicate identifier values, and
44 ambiguous overlapping year columns — each tagged `critical`, `warning`, or `info`.
456. Assemble the JSON report (`status`, file metadata, `findings`, `recommendations`,
46 `column_profiles`), print it, and clean up the temp file if one was created.
477. Relay the report to the user as-is; do not modify the source file, `datasets.json`,
48 or any other project file based on the findings — that's a separate, explicit step.
49
50## Output
51
52A single JSON object printed to stdout:
53
54- `status` — `ok`, `warning`, or `critical`.
55- `file`, `file_name`, `source_type` (`local` or `url`), `row_count`, `column_count`.
56- `findings` — structured issues, most severe first.
57- `recommendations` — de-duplicated suggested next steps.
58- `column_profiles` — per-column summary (nulls, blanks, distinct count, sample
59 values, inferred types, numeric/year ranges).
60
61No files are created or modified. A remote URL's temp download is removed on exit,
62success or failure alike.
63
64## Error Handling
65
66| Symptom | Cause | Fix |
67| --- | --- | --- |
68| `"File ... is not available."` | Local path is wrong, or the URL download failed | Verify the path or URL is reachable and retry. |
69| `"Only CSV and TSV files are supported right now."` | File extension isn't `.csv`/`.tsv` | Convert the file, or point to its tabular source instead. |
70| `"... does not contain tabular headers."` | File is empty or the header row is malformed | Open the file and confirm it has a valid, non-empty header line. |
71| Command hangs on a URL | Remote host is slow or blocks non-browser requests | Download the file manually and audit the local copy instead. |
72| `python3: command not found` | Python 3 isn't installed or not on `PATH` | Install Python 3, or run the audit where it's available. |
73| Report looks truncated in the terminal | Large report wrapped/paginated by the shell | Redirect to a file (`> report.json`) and open it separately. |
74
75## Examples
76
77### Example 1 — Audit a local CSV before publishing
78
79```
80/portaljs-check-data-quality ./public/data/trash.csv
81```
82
83### Example 2 — Audit a remote CSV over HTTPS
84
85```
86/portaljs-check-data-quality https://example.com/trash.csv
87```
88
89### Example 3 — Audit a TSV and save the report for review
90
91```bash
92bash scripts/check-data-quality.sh ./data/emissions.tsv > /tmp/emissions-quality.json
93```
94
95### Example 4 — Read a `critical` status report
96
97```json
98{
99 "status": "critical",
100 "findings": [
101 { "severity": "critical", "check": "duplicate_rows", "message": "42 duplicate rows found." }
102 ],
103 "recommendations": ["Review and deduplicate repeated rows if they are not intentional."]
104}
105```
106Fix the flagged rows/columns, then re-run the audit before publishing.
107
108## Resources
109
110- Full workflow: [`.claude/commands/portaljs-check-data-quality.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/portaljs-check-data-quality.md)
111- Detailed check catalog and troubleshooting: [`references/reference.md`](references/reference.md)
112- Related skills: `portaljs-add-dataset`, `portaljs-define-schema`
113- Python `csv` module (parsing behavior this audit relies on): <https://docs.python.org/3/library/csv.html>
datopian