Check Dataset Quality

Dataset Audit

Profile and validate a dataset for ML readiness. Identify blockers and quality issues before they cause silent model failures.

Invocation

Arguments ($ARGUMENTS) are interpreted as:

• path/to/data.csv — local CSV file
• path/to/data.parquet — local Parquet file
• path/to/data.jsonl — local JSONL file
• hf://owner/dataset-name or bare owner/dataset-name — HuggingFace Hub dataset
• path/to/images/ — image directory (expected: per-class subdirs)
• schema.table_name or database.schema.table — database table (requires --db-url)
• --db-url DSN — SQLAlchemy-compatible connection string (e.g. postgresql://user:pass@host/db, bigquery://project/dataset, sqlite:///path/to/db.sqlite)
• --db-query SQL — audit the result of an arbitrary SQL query instead of a whole table
• --split — which split(s) to load for file-based sources (default: all available)
• --text-col — primary text column for NLP tasks
• --label-col — target/label column
• --id-col — unique entity identifier for leakage detection
• --task — task type to tailor checks
• --sample N — row cap for large datasets/tables (default: full scan up to 1M rows; for DB uses TABLESAMPLE or LIMIT)
• --format — force format detection (use db to skip file detection)

Target: $ARGUMENTS

Your responsibilities

1. Load and profile

See references/format-loading.md for efficient loading per format.

Establish a quick profile before running checks:

• Row and column counts per split
• Detected format and encoding
• Column names, dtypes, and type coercion failures
• Memory footprint estimate
• Sample of first 5 rows

Report the profile before proceeding to checks.

2. Run core quality checks

Always run these regardless of task type:

Structural checks

• Schema consistency: same column names and types across train/validation/test
• Parse errors: rows that fail to load, truncated JSONL lines, corrupt Parquet row groups
• Encoding issues: mojibake, null bytes, mixed encodings in text columns

Missingness

• Per-column null rate per split
• Columns with missingness that varies significantly between splits (> 5pp delta)
• Missing labels (null or empty in --label-col)

Duplicates

• Exact row duplicates within each split
• Near-duplicate detection on text column (MinHash or length+prefix heuristic)
• --id-col duplicates within a split (entity appears twice)
• Cross-split --id-col overlap (leakage)

Label health (when --label-col provided)

• Classification: class distribution, imbalance ratio, rare classes (< 0.5%)
• Regression: target distribution, extreme skew, zero inflation, impossible values
• NLP: sequence length distribution, empty sequences, truncation risk at model max_length

Outliers and value sanity

• Numeric columns: IQR-based outlier rate, impossible values (negative age, etc.)
• Text columns: extreme length outliers (< 5 tokens or > 10× median)
• Image directories: corrupt/unreadable files, extreme aspect ratios, near-zero size

Leakage checks

• Direct leakage: --label-col value appears verbatim in text features
• Cross-split entity overlap via --id-col
• Train rows with timestamps newer than validation/test (if timestamp column detected)

Run task-specific checks from references/audit-checks.md.

Use uv run scripts/profile-dataset.py <path> ... for efficient statistical profiling when the dataset is local. For database targets, push as much computation as possible to the DB engine (COUNT, NULL rates, MIN/MAX, APPROX_COUNT_DISTINCT) rather than pulling all rows — see references/format-loading.md.

3. Severity classification

Blocker examples: cross-split leakage, missing labels in > 20% of train, schema mismatch that breaks inference, corrupt Parquet that prevents loading.

If any blocker exists, recommend NO-GO explicitly.

4. Recommend concrete remediation

Each finding must include:

• Affected columns, splits, and row counts
• Why it matters for model quality
• One or two concrete fixes with code or commands
• Re-check criterion

Prefer remediations that preserve data provenance and are reproducible.

5. Output format

Dataset Audit Report
====================
Path: <path or dataset name>
Format: <detected format>
Task: <task or unknown>

Profile:
- train: <N> rows × <M> cols  |  validation: <N> rows  |  test: <N> rows
- Columns: <list>
- Label col: <col> (<class distribution or range>)

Blockers:
1) <finding> — affects <splits/cols> — fix: <action>

High:
1) <finding> — affects <splits/cols> — fix: <action>

Medium:
1) <finding>

Low:
1) <finding>

Recommended actions (ordered by priority):
1) ...
2) ...

Decision: GO | NO-GO
Confidence: high|medium|low

JSON artifact

Write check-dataset-quality.json to --out-dir (or ./ if invoked standalone) following the schema in ../../references/schemas.md. Use vocabulary from ../../references/vocabulary.md.

Key fields to populate:

• decision: GO / NO-GO
• profile: row/column counts, label distribution
• blocker_count, high_count, medium_count, low_count
• findings: one entry per finding from the audit report with appropriate severity

6. Stop conditions

Stop when:

• A complete severity-ranked audit report is delivered with Go/No-Go
• Required clarification (unknown label column, format ambiguity) is explicitly requested
• Dataset cannot be loaded and the error is clearly reported

Quick heuristics

• Any --id-col overlap between train and test → blocker leakage
• Validation accuracy too high too fast → check leakage before trusting the number
• Schema drift between splits → silent feature-engineering bugs at inference time
• Rare class < 0.5% in classification → metric masking; use macro F1 or PR-AUC
• Empty or near-empty text fields → tokenizer produces degenerate sequences
• Class label strings that differ only in case or whitespace → silent label noise

Example

/ml-skills:check-dataset-quality hf://allenai/c4 --split train --text-col text --task language-modeling --sample 50000

Dataset Audit Report
====================
Path: allenai/c4 (HuggingFace Hub, split=train, sampled 50,000 rows)
Format: HF datasets (Arrow)
Task: language-modeling

Profile:
- train: 50,000 rows × 2 cols (text, timestamp)
- Columns: text (string), timestamp (string, parseable as datetime)
- Label col: none

Blockers:
none

High:
1) Sequence length distribution: p99=8,412 tokens (GPT-2 tokenizer). If max_length=1024, 23% of examples will be truncated to <50% of their content.

Medium:
1) 142 rows (0.28%) contain near-duplicate text (≥ 0.95 Jaccard similarity).
2) 38 rows have text length < 20 characters; likely navigation fragments.

Recommended actions (ordered by priority):
1) Use packing/concatenation (datasets.map with stride) to avoid truncation waste.
2) Filter rows with char_count < 50 before tokenization.

Decision: GO
Confidence: high

/ml-skills:check-dataset-quality ml.training_examples --db-url postgresql://user:pass@host/prod --label-col label --id-col user_id --task classification --sample 200000

Dataset Audit Report
====================
Source: postgresql://host/prod → ml.training_examples (sampled 200,000 rows via TABLESAMPLE)
Format: database table
Task: classification

Profile:
- rows: 200,000 × 14 cols
- Columns: user_id (int8), text (text), label (int2), created_at (timestamptz), ...
- Label col: label (0: 61%, 1: 39%)

Blockers:
1) 3,812 user_id values appear in both ml.training_examples and ml.test_examples — cross-table entity leakage.

High:
1) label is NULL in 4.2% of rows (8,400 rows).
2) text column has 0-character strings in 1.1% of rows.

Recommended actions (ordered by priority):
1) Rebuild train/test split with GROUP BY user_id to prevent entity overlap.
2) Filter or impute NULL labels before training.

Decision: NO-GO
Confidence: high

Additional resources

• references/audit-checks.md — Detailed check specifications with thresholds and severity criteria
• references/format-loading.md — Efficient loading patterns per format
• scripts/profile-dataset.py — Dataset profiling script (run with uv run) that outputs JSON stats