dataset-quality-audit
A data quality auditing tool that runs 12-dimension quality checks on tabular data, producing per-dimension scores (0–100), an overall grade, and actionable fix suggestions.
Capabilities
| Dimension |
Description |
| Missing Values |
Count and percentage of null/NaN values per column |
| Duplicate Rows |
Number and percentage of fully duplicated rows |
| Type Consistency |
Mixed types within a single column (e.g., numbers mixed with text) |
| Value Range / Outliers |
Outlier detection using the IQR method |
| Format Compliance |
Consistency of date, email, phone number, and other formatted fields |
| Uniqueness Constraints |
Whether ID-type columns contain duplicates |
| Whitespace Issues |
Leading/trailing spaces, empty strings, whitespace-only values |
| Constant Columns |
Columns with only a single unique value (zero information) |
| Distribution Skewness |
Whether numeric columns have excessive skewness |
| Column Naming |
Spaces, special characters, or inconsistent casing in column names |
| Cardinality Anomalies |
Unusually high or low number of unique values |
| Cross-Column Consistency |
Logical checks across columns (e.g., start date before end date) |
Quick Start
# Basic quality check
python3 scripts/data_quality_checker.py data.csv
# Save report as JSON
python3 scripts/data_quality_checker.py data.csv --output report.json
# Specify ID columns (for uniqueness checks)
python3 scripts/data_quality_checker.py users.csv --id-columns "user_id,email"
# Specify date columns (for format checks)
python3 scripts/data_quality_checker.py orders.csv --date-columns "created_at,updated_at"
Detailed Usage
Basic Invocation
python3 scripts/data_quality_checker.py <data-file> [options]
Parameters
| Parameter |
Short |
Required |
Default |
Description |
input |
— |
Yes |
— |
Path to input file (CSV/TSV/Excel/JSON) |
--output |
-o |
No |
stdout |
Path for the JSON report output |
--id-columns |
-id |
No |
Auto-detect |
Comma-separated column names that should be unique |
--date-columns |
-dc |
No |
Auto-detect |
Comma-separated column names containing dates |
--sample |
-s |
No |
All rows |
Number of rows to sample (useful for large files) |
--encoding |
-e |
No |
utf-8 |
File encoding |
Output Format (JSON)
{
"file": "data.csv",
"rows": 10000,
"columns": 15,
"overall_score": 78.5,
"grade": "B",
"dimensions": {
"missing_values": {
"score": 85.0,
"issues": [
{"column": "age", "missing_count": 150, "missing_pct": 1.5, "suggestion": "Fill with median or mode"}
]
},
"duplicates": {
"score": 95.0,
"issues": [...]
}
},
"top_suggestions": [
"Column 'age' has 1.5% missing values — consider filling with the median",
"Found 200 fully duplicated rows — consider deduplication"
]
}
Grading Scale
| Grade |
Score Range |
Meaning |
| A+ |
95–100 |
Excellent quality — ready for use as-is |
| A |
90–95 |
Good quality — minor issues only |
| B |
80–90 |
Moderate quality — recommended to fix before use |
| C |
60–80 |
Poor quality — significant cleaning required |
| D |
40–60 |
Very poor quality — many issues need attention |
| F |
0–40 |
Essentially unusable — requires re-collection or major cleanup |
Dependencies
pip install pandas numpy
1---2name: dataset-quality-audit3description: Run comprehensive quality checks on tabular data (CSV/Excel/TSV/JSON), detecting missing values, duplicates, outliers, format issues, and type inconsistencies to produce an overall score, grade, and actionable suggestions. Triggered when users ask to check data quality, find missing or duplicate values, detect outliers, validate formats, profile data, or clean data.4license: MIT5---6
7# dataset-quality-audit
8
9A data quality auditing tool that runs 12-dimension quality checks on tabular data, producing per-dimension scores (0–100), an overall grade, and actionable fix suggestions.
10
11## Capabilities
12
13| Dimension | Description |
14|-----------|-------------|
15| Missing Values | Count and percentage of null/NaN values per column |
16| Duplicate Rows | Number and percentage of fully duplicated rows |
17| Type Consistency | Mixed types within a single column (e.g., numbers mixed with text) |
18| Value Range / Outliers | Outlier detection using the IQR method |
19| Format Compliance | Consistency of date, email, phone number, and other formatted fields |
20| Uniqueness Constraints | Whether ID-type columns contain duplicates |
21| Whitespace Issues | Leading/trailing spaces, empty strings, whitespace-only values |
22| Constant Columns | Columns with only a single unique value (zero information) |
23| Distribution Skewness | Whether numeric columns have excessive skewness |
24| Column Naming | Spaces, special characters, or inconsistent casing in column names |
25| Cardinality Anomalies | Unusually high or low number of unique values |
26| Cross-Column Consistency | Logical checks across columns (e.g., start date before end date) |
27
28## Quick Start
29
30```bash
31# Basic quality check
32python3 scripts/data_quality_checker.py data.csv
33
34# Save report as JSON
35python3 scripts/data_quality_checker.py data.csv --output report.json
36
37# Specify ID columns (for uniqueness checks)
38python3 scripts/data_quality_checker.py users.csv --id-columns "user_id,email"
39
40# Specify date columns (for format checks)
41python3 scripts/data_quality_checker.py orders.csv --date-columns "created_at,updated_at"
42```
43
44## Detailed Usage
45
46### Basic Invocation
47
48```bash
49python3 scripts/data_quality_checker.py <data-file> [options]
50```
51
52### Parameters
53
54| Parameter | Short | Required | Default | Description |
55|-----------|-------|----------|---------|-------------|
56| `input` | — | Yes | — | Path to input file (CSV/TSV/Excel/JSON) |
57| `--output` | `-o` | No | stdout | Path for the JSON report output |
58| `--id-columns` | `-id` | No | Auto-detect | Comma-separated column names that should be unique |
59| `--date-columns` | `-dc` | No | Auto-detect | Comma-separated column names containing dates |
60| `--sample` | `-s` | No | All rows | Number of rows to sample (useful for large files) |
61| `--encoding` | `-e` | No | utf-8 | File encoding |
62
63## Output Format (JSON)
64
65```json
66{
67 "file": "data.csv",
68 "rows": 10000,
69 "columns": 15,
70 "overall_score": 78.5,
71 "grade": "B",
72 "dimensions": {
73 "missing_values": {
74 "score": 85.0,
75 "issues": [
76 {"column": "age", "missing_count": 150, "missing_pct": 1.5, "suggestion": "Fill with median or mode"}
77 ]
78 },
79 "duplicates": {
80 "score": 95.0,
81 "issues": [...]
82 }
83 },
84 "top_suggestions": [
85 "Column 'age' has 1.5% missing values — consider filling with the median",
86 "Found 200 fully duplicated rows — consider deduplication"
87 ]
88}
89```
90
91## Grading Scale
92
93| Grade | Score Range | Meaning |
94|-------|------------|---------|
95| A+ | 95–100 | Excellent quality — ready for use as-is |
96| A | 90–95 | Good quality — minor issues only |
97| B | 80–90 | Moderate quality — recommended to fix before use |
98| C | 60–80 | Poor quality — significant cleaning required |
99| D | 40–60 | Very poor quality — many issues need attention |
100| F | 0–40 | Essentially unusable — requires re-collection or major cleanup |
101
102## Dependencies
103
104- Python 3.8+
105- pandas
106- numpy
107
108```bash
109pip install pandas numpy
110```
majiayu000