# Data Quality

> Quality dimensions quick reference and remediation decision tree for tabular data assessment

- Skill: `dathere/data-quality` (Agent Skill)
- Install (CLI): `npx skillmds@latest add dathere/data-quality`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dathere/data-quality/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: dathere (https://skillmd.com/u/dathere)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/dathere/data-quality

---


# Data Quality Assessment with qsv

For the full step-by-step profiling workflow, use the `/data-profile` command. This skill provides quick-reference guidance for quality assessment and remediation decisions.

## Quality Dimensions (Quick Reference)

| Dimension | Key Question | Primary Check | Red Flag |
|-----------|-------------|---------------|----------|
| **Completeness** | Missing values? | `stats` — `nullcount`, `sparsity` | Sparsity > 0.5 |
| **Uniqueness** | Unwanted duplicates? | `stats --cardinality` vs row count | Key column cardinality < row count |
| **Validity** | Correct formats/types? | `stats` — `type`; `validate schema.json` | String type on numeric column |
| **Consistency** | Uniform formats? | `frequency` — case variants; `sniff` — encoding | Same value in different cases |
| **Accuracy** | Plausible values? | `stats` — min/max/stddev | Values > 3 stddev from mean |
| **Column Name Quality** | Headers safe & descriptive? | `safenames --verify` | Spaces, special chars, or duplicates in headers |
| **Conformity** | Values follow standards? | `searchset` with domain regex | Non-standard codes (country, state, zip, phone) |
| **Referential Integrity** | Foreign keys valid? | `joinp --left-anti` | Orphaned references across related files |
| **Injection Safety** | Malicious payloads? | `searchset` with injection regex | Formula/SQL injection patterns in cells |
| **Documentation** | Dataset described? | `describegpt --all` | No Data Dictionary or Description |

## Remediation Decision Tree

When a quality issue is found, choose the right fix:

| Problem | Severity | Fix Command | When to Skip |
|---------|----------|-------------|-------------|
| Ragged rows | High | `fixlengths` | Never — breaks downstream tools |
| Wrong encoding | High | `input` | File is already UTF-8 (check with `sniff`) |
| Unsafe column names | Medium | `safenames` | Headers already safe (no spaces/special chars) |
| Leading/trailing whitespace | Medium | `sqlp` with `TRIM(col)` | Stats show no difference between `min`/`max` lengths and trimmed values |
| Duplicate rows | Medium | `dedup` (or `extdedup` for >1GB) | `stats --cardinality` on key columns shows all unique |
| Inconsistent case | Low | `sqlp` with `UPPER(col)` or `LOWER(col)` | `frequency` shows no case variants |
| Empty values | Low | `sqlp` with `COALESCE(NULLIF(col, ''), 'N/A')` | Nulls are semantically meaningful |
| Non-conforming values | Medium | `searchset` + `search --flag` | No domain standard applies |
| Orphaned foreign keys | Medium | `joinp --left-anti` | Single-file dataset with no references |
| Injection payloads | High | `searchset` with injection regex + sanitize | Data is internal-only and never opened in spreadsheets or loaded into databases |
| Invalid rows | Low | `validate schema.json` + filter | No schema available |

## Fix Ordering

Always apply fixes in this order to avoid cascading issues:

```
1. input          (encoding — must be UTF-8 before anything else)
2. safenames      (headers — fixes names before column references)
3. fixlengths     (structure — ensures consistent field counts)
4. sqlp with TRIM()    (whitespace — clean values before dedup)
5. dedup          (duplicates — remove after trimming so "foo " and "foo" match)
6. validate       (validation — check against schema last)
```

## Stats Cache as Quality Data Schematic

After running `stats --cardinality --stats-jsonl` (basic moarstats auto-runs), read the `.stats.csv` cache to assess quality in one pass:

| Cache Column | Quality Signal |
|-------------|----------------|
| `nullcount` | Completeness — 0 is ideal |
| `sparsity` | Completeness — ratio of nulls (0.0–1.0) |
| `cardinality` | Uniqueness — compare to row count |
| `type` | Validity — check expected types |
| `min` / `max` | Accuracy — plausible range? |
| `mean` / `stddev` | Accuracy — outlier detection (>3σ) |
| `outliers_total_cnt` | Accuracy — from moarstats; outlier count per column |
| `mode` | Consistency — dominant value expected? |

### Advanced Stats (via `moarstats --advanced`)

Run `moarstats --advanced` to enrich the cache with distribution shape metrics:

| Cache Column | Quality Signal |
|-------------|----------------|
| `kurtosis` | >3 heavy tails (outlier-prone), <3 light tails; >10 = extreme outliers |
| `bimodality_coefficient` | >=0.555 suggests bimodal distribution (possible mixed populations) |
| `jarque_bera_pvalue` | <0.05 = NOT normally distributed; flag analyses assuming normality |
| `gini_coefficient` | Near 1 = extreme concentration; near 0 = uniform |
| `shannon_entropy` | Low = concentrated values; high = diverse |
| `winsorized_mean` | Compare to `mean` — large difference signals outlier influence |
| `median_mean_ratio` | <0.8 or >1.2 = significantly skewed; mean may be misleading |
| `range_stddev_ratio` | Very high = extreme outliers relative to variability |
| `cv` | >100% = high relative variability; data is highly spread relative to mean |
| `mad_stddev_ratio` | >0.8 = stddev is reliable; <<0.8 = outliers inflating stddev |
| `mode_zscore` | Far from 0 = mode is atypical; possible mixed populations |
| `trimean` | Robust central tendency: (Q1 + 2*median + Q3)/4; compare to mean for skew detection |
| `midhinge` | Midpoint of middle 50%: (Q1+Q3)/2; robust center measure |
| `robust_cv` | MAD/abs(median); outlier-resistant coefficient of variation |
| `theil_index` | Inequality measure (0=equal); decomposable into within/between group; only for positive values |
| `mean_ad` | Average absolute distance from mean; less sensitive to outliers than stddev |
| `simpsons_diversity_index` | Probability two random values differ (0-1); more intuitive than entropy |