Smart Eda

Smart EDA

Adaptive exploratory data analysis for tabular data. The skill profiles the dataset first, then picks methods based on the data's actual shape, types, scale, and the user's intent.

Core philosophy

EDA is detective work, not a checklist. John Tukey introduced it as "actively incisive, rather than passively descriptive, with real emphasis on the discovery of the unexpected." The skill operationalizes three attitudes:

• Profile before analyzing. Never apply a method without first checking whether the data fits its assumptions.
• Adapt to the data. A dtype=int64 column with 4 unique values is a categorical, not a numeric. A correlation of 0.9 between two columns might be a leak, not a finding. Branch on what's actually there.
• Surface insights, not just stats. Every EDA closes with top findings, anomalies, hypotheses, and concrete next steps — not a wall of describe() output.

When to use which tier

EDA work scales. The skill operates in three tiers; pick based on the request and the data:

• Tier 1 — 30-second scan. Shape, dtypes, missing percent, dupes, sample. Use when the user just says "what's in this" or as the warmup for any deeper work.
• Tier 2 — Standard EDA (default). Tier 1 + per-column univariate (adapted to type) + correlation + target relationship if a target column exists + time-series plot if datetime found + quality issues + top findings. Use this unless the request explicitly asks for less or more.
• Tier 3 — Deep dive. Tier 2 + formal tests with effect sizes + multivariate outliers + PCA/clustering + change-point detection + hypothesis catalog. Use when the user asks for "thorough", "deep", "everything", "full audit", or when Tier 2 surfaces something that demands closer inspection.

State the chosen tier briefly at the start, e.g. "Running standard EDA — I'll go deeper on anything that looks worth it."

Workflow

Phase 0 — Understand the ask and locate the data

Before any code, answer:

1. What is the user asking? Quick scan, full EDA, specific question (e.g. "find outliers"), or open-ended ("look at this")?
2. Where is the data? Uploaded file in /mnt/user-data/uploads/, a path in their message, or a DataFrame mentioned in prior context?
3. One file or many? If multiple files were uploaded or referenced, this is a multi-file EDA — see Phase 1.5. Common patterns: relational tables (orders + customers + products), monthly/regional snapshots of the same schema, or independent datasets the user wants compared.
4. Is there a target/outcome variable? A column named like target, label, y, outcome, churn, price, success, or an explicit mention. If so, EDA will pivot toward feature-vs-target analysis.
5. Is there a domain hint? "Sales", "patients", "trades", "logs" change which patterns to look for and which quality issues to flag.

If the user uploaded a file with no instruction, default to Tier 2 and proceed. Don't ask 5 clarifying questions — that's the fastest way to make the skill annoying. Asking one question is fine if there's genuine ambiguity (e.g. "Quick look or full EDA?"); otherwise just start and let them redirect.

Phase 1 — Load and run reconnaissance

Use the bundled scripts/profile_data.py to do this efficiently. It handles CSV, TSV, Excel, Parquet, and JSON-lines, infers types, and emits a structured profile — without burning tokens on Claude writing this boilerplate every time:

python /path/to/skill/scripts/profile_data.py <data-file> --output /home/claude/profile.json

The profile includes: shape, memory, per-column dtype, semantic type (numeric/categorical/datetime/text/id/boolean), nunique, missing count, sample values, and the basic stats. Read the JSON, then narrate the recon in plain language — don't dump the JSON to the user.

If the file is unusual (>500MB, weird format, multiple sheets), fall back to manual loading with size-aware techniques (see references/large_data.md).

Phase 1.5 — Multi-file analysis (when applicable)

When the user has more than one tabular file, use scripts/multi_file_eda.py. It profiles each file, detects join keys via name/type compatibility and value overlap, infers cardinality (1:1 / 1:N / N:N), measures orphan rates on both sides of every relationship, and produces a single dashboard showing the schema, the file inventory, the detected relationships, and a combined EDA on the auto-joined data.

python /path/to/skill/scripts/multi_file_eda.py file1.csv file2.csv file3.csv \
    --output-dir /mnt/user-data/outputs --target sales

The script:

• Profiles each file individually (per-file quality flags, types, etc.)
• Looks for shared column names (or singular/plural pairs like customer ↔ customers) with compatible types and ≥30% value overlap
• Picks the strongest non-conflicting relationships and performs left-joins, starting from the fact table (the one with the most outgoing N:1 references)
• Runs the standard eda_runner flow on the joined dataset and embeds the results
• Renders a schema diagram with the fact table centered and dimension tables fanning out, with arrows labeled by cardinality

When to use which:

• Single file → standard flow (profile_data.py → eda_runner.py)
• Multiple files, related (orders/customers/products) → multi-file EDA, auto-join
• Multiple files, same schema (monthly snapshots) → concatenate first with pd.concat, then standard flow on the concatenated frame; mention the concat to the user
• Multiple files, unrelated → run standard flow on each separately, present each dashboard

If the multi-file script reports zero relationships and the files are clearly unrelated (e.g., different domains), fall back to per-file analysis. If it reports zero but you suspect there should be relationships (column names differ but data matches), tell the user — they may need to rename a column or specify the keys manually.

Phase 2 — Data quality scan

Catch the issues that derail later analysis. From the profile, surface:

• Missing values — count + percent per column. Flag any > 5% as worth discussing; > 50% as probably-drop candidates.
• Duplicates — full-row, and key-based if an obvious ID column exists.
• Constants / near-constants — nunique() == 1 or top frequency > 99%.
• Mixed types within a column — object columns where values aren't all the same Python type.
• Whitespace / casing inconsistency — for object columns, compare nunique() before and after .str.strip().str.lower().
• Out-of-range / impossible values — age < 0, future birthdates, negative quantities, percentages > 100.
• Encoding artifacts — characters like Ã, ’, \xa0 in text columns.

Don't fix these yet. The job at this phase is to find them and decide whether they affect later analysis. For deep audits, see references/quality_issues.md.

Phase 3 — Univariate analysis (branch by effective type)

For each column, decide its effective type (which may differ from its dtype) and apply the right methods. The decision rules:

• dtype is numeric AND nunique ≤ 10 → treat as ordinal-categorical: bar chart, value counts, not histogram
• dtype is numeric AND nunique > 10 → numeric: histogram + KDE, boxplot, 5-number summary, skewness, IQR-outlier flag
• dtype is object AND all values match a date/time pattern → datetime: parse and treat as datetime
• dtype is object AND mean string length > 50 → free text: length distribution, top n-grams, language/encoding check
• dtype is object AND nunique / total > 0.95 → ID-like: skip distribution analysis, just confirm uniqueness
• dtype is object otherwise → categorical: value counts, bar chart, rare-category flag (<1%), entropy
• dtype is datetime → range, granularity, gap detection, frequency by year/month/dow/hour, seasonality hint
• dtype is bool → ratio of True/False

For numeric columns, always:

1. Report the 5-number summary, mean, std, skewness
2. If |skewness| > 1, mention that a log/Box-Cox transform may help and suggest it
3. Compute IQR-based outlier count; if > 1% of rows, mention it but don't drop anything

For categorical columns, always:

1. Report top-5 values with frequencies
2. Flag high cardinality (nunique > 50) as needing binning before modeling
3. Flag rare categories (< 1% frequency) as candidates for "Other"

The exhaustive method catalog by type is in references/method_catalog.md — consult it when the data has unusual columns.

Phase 4 — Relationships (bivariate and multivariate)

Run this whenever there are ≥ 2 columns. Be selective: don't generate 100 pair plots on a 50-column dataset.

If a target column was identified in Phase 0:

• Numeric features vs. numeric target: scatter + LOWESS, Pearson + Spearman, log-scale check
• Categorical features vs. numeric target: boxplot/violin grouped, group means with 95% CI, ANOVA (with Welch's correction for unequal variance) or Kruskal-Wallis if non-normal, effect size (η²)
• Numeric features vs. categorical target: distribution per class (overlaid KDE or violin), Mann-Whitney / Kruskal-Wallis, point-biserial correlation
• Categorical features vs. categorical target: contingency table, chi-square (or Fisher's if expected < 5), Cramer's V

Always report effect size alongside p-values. p-values without effect sizes are misleading — a tiny effect is "significant" with enough data and uninteresting.

If no target:

• Numeric × numeric: Pearson + Spearman correlation matrix, heatmap. Flag |r| > 0.9 as possible redundancy / leakage.
• Categorical × categorical: Cramer's V matrix for the top categorical pairs.
• Don't try to compute correlation between every pair if there are > 30 columns; focus on either (a) pairs with the highest variance or (b) the top correlations.

Multivariate:

• Pair plot if ≤ 8 numeric columns AND ≤ 5k rows (otherwise skip — the plot becomes unreadable)
• VIF for multicollinearity if numeric columns ≥ 5 and a target was given
• PCA with scree plot if numeric columns ≥ 10

For statistical-test interpretation guidance (when each test is appropriate, what its assumptions are, how to report it), see references/statistical_tests.md.

Phase 5 — Specialized analysis

Run only what applies to this data:

• Time series — if a datetime column was detected and the data appears time-ordered: line plot, resample to a sensible frequency, rolling mean + std, decomposition (trend / seasonal / residual), stationarity test (ADF), ACF/PACF. See references/time_series.md.
• Class imbalance — if a categorical target with ratio < 1:5 between classes, flag it explicitly with implications for modeling.
• Drift / split comparison — if the user has train and test data, run distribution comparisons (Kolmogorov-Smirnov per feature) and flag any drift.

Phase 6 — Synthesize insights

This is the deliverable, not the stats. Produce four lists, in this order:

1. Top findings — 3 to 7 bullet points stating the most interesting things the data shows, in plain language. Lead with the most surprising or actionable.
2. Quality issues to address — concrete problems the user should fix or decide on (which missing-value strategy, which outliers to investigate, which columns to drop).
3. Hypotheses worth testing — testable claims the EDA suggested. "Customers from segment A churn more than B" not "there appears to be variation across segments."
4. Recommended next steps — what to do with this data: clean these things, transform these columns, model with these features, collect more of this.

A finding without a "so what?" isn't a finding.

Phase 7 — Deliverables

The skill produces two complementary primary deliverables — one visual, one textual — plus the reproducible script. They serve different reading modes and don't duplicate each other.

By default produce:

• dashboard.html — the visual deliverable. Self-contained interactive HTML, ~5MB with Plotly inline (works offline, opens in any browser). Provides the high-level overview:

  • 4 adaptive KPI cards
  • Hero time-series chart with annotated peak/low (or top categorical if no datetime)
  • Target distribution with skew note + correlation heatmap
  • Two top categorical breakdowns (auto-selected for highest discriminating power)
  • Quality-flags banner with severity-color dots
  • Insights section: top findings, hypotheses worth testing, recommended next steps
  • Hover tooltips, drag-to-zoom range slider on the time series

• eda_summary.md — the detailed analytical report. Multi-page markdown with the specific numbers: per-column statistics tables (count, mean, median, std, min/max, quantiles, skew, kurtosis, outliers for every numeric; top values with counts and percentages for every categorical; range and granularity for every datetime), full Pearson correlation matrix, ranked correlation pairs with leakage-flag interpretation, target-feature analysis (correlations for numeric, group-mean discrimination ratios for categorical), the auto-generated findings/hypotheses/steps with their supporting numbers, and a methodology section listing what was run and what wasn't. Designed to be grep-able, diff-able, and attachable to a PR or quarterly review.

• eda_<dataset_name>.py — reproducible Python script. Separate deliverable. The user can run, modify, version-control, or commit this script to recreate every chart and finding from the dashboard.

• Per-column plots (univariate distributions, target-relationship plots, correlation heatmap PNGs in the same folder) — supporting material referenced by the dashboard and the markdown.

• For Tier 3 deep dives: also produce an HTML report using the helper script with formal tests, multivariate outliers, PCA, change-point detection.

The two primary deliverables are designed for different audiences:

• The dashboard is for the person who wants to understand the data — managers, decision-makers, anyone scanning for the takeaway
• The markdown is for the person who needs to verify or document the analysis — analysts reviewing the work, reviewers signing off, future-you reading this in three months

Use present_files to surface both. Lead with dashboard.html for users who'll open it in a browser; on text-only surfaces, lead with eda_summary.md. The two complement each other — the dashboard lacks the per-column statistics tables; the markdown lacks the visual pattern-recognition.

Adaptive selection — quick reference

Working principles

• Use the bundled scripts. scripts/profile_data.py, scripts/eda_runner.py, and scripts/dashboard_html.py exist precisely so Claude doesn't waste tokens rewriting the same boilerplate every time. Read them first; extend them if needed; don't duplicate them.
• Optional dependencies. The interactive HTML dashboard requires plotly (>= 5.0). If it's not installed and the user wants the interactive output, run pip install plotly (use --break-system-packages in container environments). The runner falls back gracefully — if Plotly isn't available, only the static PNG dashboard is built and a note is printed.
• Save plots before showing them. Always write to /mnt/user-data/outputs/eda_plots/ so they survive the conversation.
• Cite assumptions. When you use Pearson, note that it assumes linearity. When you use a t-test, note that it assumes normality and equal variance. When you sample, say so. The user shouldn't need to second-guess the analysis.
• Be honest about uncertainty. If a finding could be a coincidence (small sample, weak effect), say so. If a correlation could be a confounder, say so.
• Don't drop or impute silently. Flag the issues; let the user decide. The skill's job is to inform, not to clean for them (unless they explicitly ask).
• One question is fine, five is not. If the data is ambiguous about intent or target, one short clarifying question is OK. More than that wastes the user's time.

When to reach for the reference files

The reference files contain the exhaustive material. Read them when needed — don't try to remember everything in the SKILL.md.