gabrielmoreira Data Analysis
| Property | Value |
|---|---|
| Domain | Data Analytics |
| Category | Analysis & Insight Extraction |
| Components | SKILL.md + data-analysis.instructions.md + analyze.prompt.md |
| Depends | data-visualization (chart output), data-ingest.cjs (ingestion) |
Overview
Turn raw data into actionable insight statements. This skill covers the full EDA pipeline: profiling the dataset, exploring distributions, finding correlations, detecting anomalies, and -- critically -- translating statistical findings into business-language narratives tagged with story intents for downstream visualization.
The cardinal rule: statistics are not insights. "Mean revenue is $4.2M" is a statistic. "Revenue grew 34% YoY but growth is decelerating -- Q3 peak was 8% vs. 22% last year" is an insight.
Module 1: Data Profiling
First pass on any dataset. Compute before exploring.
| Metric | What It Tells You | Red Flag |
|---|---|---|
| Row count | Dataset scale | <100 rows limits statistical power |
| Column count | Dimensionality | >50 columns suggests feature bloat |
| Null percentage | Data completeness | >20% nulls in key column = unreliable |
| Unique count | Cardinality | Unique count = row count → likely ID |
| Type inference | String/number/date/boolean | Mixed types in same column = dirty |
| Memory estimate | Processing feasibility | >500MB warns for browser context |
| Duplicate rows | Data quality | >1% duplicates needs dedup decision |
Profiling Output Template
Dataset: {name} ({rowCount} rows x {colCount} columns)
Source: {source} | Format: {format} | Encoding: {encoding}
Column Summary:
{name}: {type} | {nullPct}% null | {uniqueCount} unique | min={min} max={max} mean={mean}
...
Quality Score: {score}/100
- Completeness: {completeness}% (columns with <5% nulls)
- Consistency: {consistency}% (columns with single type)
- Uniqueness: {uniqueness}% (no unexpected duplicates)
Warnings:
- Column "X" has 23% nulls -- consider imputation or exclusion
- Column "Y" has mixed types (78% number, 22% string) -- needs cleaning
Module 2: Descriptive Statistics
What to compute first for every numeric column.
| Statistic | When It Matters |
|---|---|
| Mean vs. Median | If mean >> median, right-skewed (outliers pull up) |
| Standard Dev | Spread -- is the data tight or dispersed? |
| Min / Max | Range -- any impossible values? |
| Percentiles | P25, P50, P75 -- where does the bulk sit? |
| Skewness | >1 or <-1 suggests non-normal distribution |
| Kurtosis | >3 = heavy tails (more outliers than expected) |
Rule of Thumb: Mean vs. Median
- If
|mean - median| / median > 0.1(10%), report median as the "typical" value - Always report both -- the gap itself is an insight
Module 3: Distribution Analysis
| Shape | What It Suggests | Story Intent |
|---|---|---|
| Normal | Stable process, predictable | Distribution |
| Right-skewed | Many small values, few large (income) | Deviation |
| Left-skewed | Most values high, some low (test scores) | Deviation |
| Bimodal | Two populations mixed together | Compare (groups) |
| Uniform | No pattern -- random or categorical codes | None (check) |
| Power law | Few items dominate (Pareto, web traffic) | Part-to-Whole |
Normality Quick Check
- Compare mean to median (>10% gap = non-normal)
- Check skewness (|skew| > 1 = non-normal)
- If important: Shapiro-Wilk test (n < 5000) or Anderson-Darling
Module 4: Correlation & Relationship
| Strength | |r| Range | Interpretation |
| -------- | ----------- | --------------------------- |
| Strong | 0.7 -- 1.0 | Likely meaningful |
| Moderate | 0.4 -- 0.7 | Worth investigating |
| Weak | 0.1 -- 0.4 | Unlikely actionable alone |
| None | < 0.1 | No linear relationship |
Simpson's Paradox Awareness
Always check: does the correlation reverse when you split by a categorical variable?
Overall: Ad spend positively correlates with sales (+0.6)
By region: In 3 of 4 regions, correlation is NEGATIVE
Cause: High-spend region has higher baseline sales (confound)
Rule: If a strong correlation exists, segment by the top 2-3 categorical variables and re-check.
Module 5: Segmentation
Group-by patterns for discovering sub-populations.
| Technique | When to Use | Output |
|---|---|---|
| Group-by aggregate | Categorical × numeric | Segment averages |
| Percentile buckets | Continuous variable, create tiers | Low/Mid/High |
| RFM analysis | Customer behavior (recency, freq, $) | Customer segments |
| Cohort analysis | Time-based grouping (signup month) | Retention curves |
| Cross-tabulation | Two categorical variables | Contingency table |
"So What?" for Segments
For each segment found, answer: "If I could only act on ONE segment, which one and why?"
Module 6: Time-Series Decomposition
| Component | What It Is | Detection |
|---|---|---|
| Trend | Long-term direction | Rolling average (window = period) |
| Seasonality | Repeating pattern at fixed intervals | Autocorrelation at lag = period |
| Residual | What's left (noise + anomalies) | Original - trend - seasonality |
Rolling Average Windows
| Data Frequency | Window |
|---|---|
| Daily | 7 or 30 |
| Weekly | 4 or 13 |
| Monthly | 3 or 12 |
| Quarterly | 4 |
Module 7: Anomaly Detection
| Method | Best For | Threshold |
|---|---|---|
| Z-score | Normal-ish distributions | |
| IQR fence | Skewed distributions | < Q1-1.5×IQR or > Q3+1.5×IQR |
| Isolation | Multivariate outliers | Score > 0.7 (heuristic) |
| Visual | Any -- always plot first | Inspect scatter/box |
Anomaly Protocol
- Detect and flag -- never auto-remove
- Investigate: is it a data error, a real outlier, or a different population?
- Document the decision: kept (real), removed (error), or separated (sub-population)
Module 8: "So What?" Translation (DIKW)
The most important module. Convert statistics into business language.
| Level | Example (Bad) | Example (Good) |
|---|---|---|
| Data | "Column revenue has 5000 values" | (Don't report raw data) |
| Information | "Mean revenue is $4.2M" | "Average quarterly revenue is $4.2M across 8 quarters" |
| Knowledge | "Revenue has a positive trend" | "Revenue grew 34% YoY but growth rate decelerated from 22% to 8%" |
| Wisdom | (Requires domain context) | "Growth is decelerating -- if Q3 seasonal effect weakens, plan for flat" |
Insight Statement Template
[WHAT]: {metric} is {value/behavior}
[SO WHAT]: This means {business implication}
[NOW WHAT]: Consider {action or follow-up question}
[STORY INTENT]: {compare|trend|deviation|distribution|relationship|part-to-whole|flow|hierarchy|spatial}
[CHART]: {recommended chart type} because {rationale}
Example Insight Statements
[WHAT]: California's population (39M) is 5x the median state (7.5M)
[SO WHAT]: Resource allocation models using state averages will dramatically under-serve CA
[NOW WHAT]: Segment by population tier, not just state count
[STORY INTENT]: Compare
[CHART]: Horizontal bar (sorted descending) because ranking 50 items with long labels
[WHAT]: Support ticket resolution time has a bimodal distribution (peaks at 2h and 48h)
[SO WHAT]: Two distinct processes exist -- quick fixes and escalated investigations
[NOW WHAT]: Separate the two populations before setting SLA targets
[STORY INTENT]: Distribution
[CHART]: Histogram with two highlighted peaks, or violin plot for visual impact
Module 9: Hypothesis Framework
Structured approach to moving from observation to testable claim.
| Step | Action |
|---|---|
| Observe | Note a pattern in the data |
| Hypothesize | State a testable claim ("X causes Y because Z") |
| Test | Check against data (filter, segment, correlate) |
| Conclude | Supported, refuted, or inconclusive |
| Narrate | Write the finding as an insight statement |
Anti-Patterns
| Anti-Pattern | Problem | Fix |
|---|---|---|
| Reporting stats only | No business meaning | Always add "so what?" |
| Correlation = causation | Misleading conclusions | Check for confounders, state "correlates" |
| Ignoring base rates | Percentages without context | Always report denominator |
| Survivorship bias | Only analyzing what's visible | Ask "what's missing from this data?" |
| Over-aggregation | Hiding variation with averages | Show distribution, not just mean |
| Premature optimization | Jumping to solutions before understanding | Complete the EDA before recommending action |