data-statistical-analysis
When to use
Use this when a decision rides on whether an effect is signal or noise: "is this difference real?", "are these correlated?", "is the trend significant?", "which points are outliers?" Reach for it instead of eyeballing a chart.
Not for: the raw business answer without rigor (use data-analyze), profiling data quality (use data-explore), or building the chart itself (use data-create-viz).
Method
- Frame the question statistically: estimation, comparison, association, or trend. State the population and unit of analysis.
- Describe first. Report n, center (mean/median), spread (SD/IQR), and shape. Visualize the distribution before testing.
- Choose the test by data type and assumptions:
- Two-group means → t-test (Welch if variances differ); non-normal/small → Mann-Whitney.
2 groups → ANOVA / Kruskal-Wallis.
- Categorical association → chi-square / Fisher (small cells).
- Numeric association → Pearson (linear) or Spearman (monotonic).
- Trend → regression, test the slope.
Decision point: check normality, independence, and variance; if an assumption fails, drop to the nonparametric option.
- Report effect size and confidence interval, not just p. A significant p with a trivial effect is not actionable.
- Outliers: detect (IQR or z-score), investigate cause, decide keep/exclude with justification — never drop silently.
- Correct for multiple comparisons (Bonferroni/BH) when running many tests.
Example
A/B test: control 4.0% conversion (n=5,000), variant 4.6% (n=5,000). Two-proportion z-test → p=0.03. But the 95% CI on the lift is [+0.05pp, +1.15pp] — the effect could be near-zero. Verdict: statistically significant, practically marginal; ship only if the +0.6pp point estimate clears the cost of the change.
Pitfalls
- Reporting p<0.05 with no effect size, implying an important result from a trivial one.
- Running a t-test on skewed or tiny samples where its normality assumption fails.
- Reading correlation as causation, ignoring a confounder.
- Testing 20 metrics, celebrating the one that hits p<0.05, and not correcting for multiplicity.
Output format
Question: hypothesis in plain + statistical terms
Descriptives: n, center, spread, distribution shape
Test: method + why + assumption checks
Result: statistic, p-value, effect size, confidence interval
Interpretation: what it means for the decision, plain language
Caveats: sample size, violated assumptions, confounders
Distinguish statistical from practical significance. Say when the sample is too small to conclude anything.
1---2name: data-statistical-analysis3description: Run descriptive stats, trend, outlier, correlation, and hypothesis tests with correct method selection and honest effect sizes.4---56# data-statistical-analysis78## When to use910Use this when a decision rides on whether an effect is signal or noise: "is this difference real?", "are these correlated?", "is the trend significant?", "which points are outliers?" Reach for it instead of eyeballing a chart.1112**Not for:** the raw business answer without rigor (use data-analyze), profiling data quality (use data-explore), or building the chart itself (use data-create-viz).1314## Method15161. **Frame the question statistically:** estimation, comparison, association, or trend. State the population and unit of analysis.172. **Describe first.** Report n, center (mean/median), spread (SD/IQR), and shape. Visualize the distribution before testing.183. **Choose the test by data type and assumptions:**19 - Two-group means → t-test (Welch if variances differ); non-normal/small → Mann-Whitney.20 - >2 groups → ANOVA / Kruskal-Wallis.21 - Categorical association → chi-square / Fisher (small cells).22 - Numeric association → Pearson (linear) or Spearman (monotonic).23 - Trend → regression, test the slope.24 Decision point: check normality, independence, and variance; if an assumption fails, drop to the nonparametric option.254. **Report effect size and confidence interval,** not just p. A significant p with a trivial effect is not actionable.265. **Outliers:** detect (IQR or z-score), investigate cause, decide keep/exclude with justification — never drop silently.276. **Correct for multiple comparisons** (Bonferroni/BH) when running many tests.2829## Example3031A/B test: control 4.0% conversion (n=5,000), variant 4.6% (n=5,000). Two-proportion z-test → p=0.03. But the 95% CI on the lift is [+0.05pp, +1.15pp] — the effect could be near-zero. Verdict: statistically significant, practically marginal; ship only if the +0.6pp point estimate clears the cost of the change.3233## Pitfalls3435- Reporting p<0.05 with no effect size, implying an important result from a trivial one.36- Running a t-test on skewed or tiny samples where its normality assumption fails.37- Reading correlation as causation, ignoring a confounder.38- Testing 20 metrics, celebrating the one that hits p<0.05, and not correcting for multiplicity.3940## Output format4142```43Question: hypothesis in plain + statistical terms44Descriptives: n, center, spread, distribution shape45Test: method + why + assumption checks46Result: statistic, p-value, effect size, confidence interval47Interpretation: what it means for the decision, plain language48Caveats: sample size, violated assumptions, confounders49```5051Distinguish statistical from practical significance. Say when the sample is too small to conclude anything.
Matrixx0070