Data Analysis

Data Analysis

Guidance for exploratory data analysis and statistical testing.

When to Use

• Exploring new datasets
• Selecting appropriate statistical tests
• Performing power analysis
• Reporting results in papers
• Validating experimental results

Exploratory Data Analysis (EDA)

EDA Workflow

1. Load and Inspect: Basic data structure
2. Summarize: Descriptive statistics
3. Visualize: Distributions and relationships
4. Identify Issues: Missing data, outliers
5. Document Findings: Key insights

Initial Inspection

# Basic checks
df.shape          # Dimensions
df.dtypes         # Data types
df.head()         # First rows
df.describe()     # Summary stats
df.isnull().sum() # Missing values

Key Questions to Answer

Visualization Guide

Statistical Test Selection

Decision Tree

Question: What are you trying to do?
│
├─ Compare groups
│   │
│   ├─ How many groups?
│   │   ├─ 2 groups → See "Two Group Comparisons"
│   │   └─ 3+ groups → See "Multiple Group Comparisons"
│   │
│   └─ Related or independent?
│       ├─ Independent (different subjects)
│       └─ Related (same subjects, before/after)
│
├─ Examine relationships
│   ├─ Two variables → Correlation, regression
│   └─ Multiple variables → Multiple regression
│
└─ Test proportions
    └─ Chi-square test

Two Group Comparisons

Multiple Group Comparisons

Checking Assumptions

Normality Tests:

• Shapiro-Wilk (n < 50)
• Kolmogorov-Smirnov (n ≥ 50)
• Visual: Q-Q plot

Homogeneity of Variance:

• Levene's test
• Visual: Box plots by group

Independence:

• By experimental design
• Durbin-Watson (for residuals)

When Assumptions Fail

Effect Sizes

Why Effect Sizes Matter

• p-values tell you if effect exists, not how big
• Effect sizes quantify the magnitude
• Required for power analysis
• Better for meta-analysis

Common Effect Sizes

Computing Effect Sizes

# Cohen's d for two groups
import numpy as np

def cohens_d(group1, group2):
    n1, n2 = len(group1), len(group2)
    var1, var2 = np.var(group1, ddof=1), np.var(group2, ddof=1)
    pooled_std = np.sqrt(((n1-1)*var1 + (n2-1)*var2) / (n1+n2-2))
    return (np.mean(group1) - np.mean(group2)) / pooled_std

Power Analysis

Key Concepts

Power Analysis Uses

1. A priori: Before study, determine needed sample size
2. Post hoc: After study, calculate achieved power
3. Sensitivity: Given n and power, what effect detectable?

Sample Size Calculation

from statsmodels.stats.power import TTestIndPower

analysis = TTestIndPower()

# Sample size for t-test
n = analysis.solve_power(
    effect_size=0.5,    # Cohen's d
    alpha=0.05,         # Significance level
    power=0.80,         # Desired power
    ratio=1.0,          # n2/n1
    alternative='two-sided'
)

Power Guidelines

Multiple Comparisons

The Problem

• Each test has α chance of false positive
• Multiple tests inflate false positive rate
• Family-wise error rate: 1-(1-α)^n

Correction Methods

Applying Corrections

from scipy import stats
import numpy as np

# Bonferroni
adjusted_alpha = 0.05 / num_tests

# Holm-Bonferroni
from statsmodels.stats.multitest import multipletests
reject, pvals_corrected, _, _ = multipletests(pvals, method='holm')

# Benjamini-Hochberg (FDR)
reject, pvals_corrected, _, _ = multipletests(pvals, method='fdr_bh')

Reporting Results

APA Format

t-test:

t(df) = X.XX, p = .XXX, d = X.XX

Example: t(45) = 2.34, p = .023, d = 0.68

ANOVA:

F(df1, df2) = X.XX, p = .XXX, η² = .XX

Example: F(2, 87) = 4.56, p = .013, η² = .095

Correlation:

r(df) = .XX, p = .XXX

Example: r(48) = .42, p = .003

Chi-square:

χ²(df, N = n) = X.XX, p = .XXX

Example: χ²(2, N = 150) = 6.78, p = .034

Reporting Guidelines

Do:

• Report exact p-values (not just < .05)
• Include effect sizes
• Report confidence intervals
• Describe what was tested

Don't:

• Say "proved" or "significant difference" alone
• Report only significant results
• Cherry-pick tests
• Over-interpret p = .049 vs p = .051

Results Table Format

Table 1: Comparison of Methods on Benchmark X

Method      Mean (SD)       95% CI          Effect Size
------------------------------------------------------
Baseline    75.2 (3.4)     [73.1, 77.3]    -
Method A    78.9 (2.8)*    [77.2, 80.6]    d = 0.52
Method B    81.4 (3.1)**   [79.5, 83.3]    d = 0.89

Note: * p < .05, ** p < .01 vs Baseline

Common Pitfalls

Quality Checklist

• Research questions clearly stated
• Appropriate test selected and justified
• Assumptions checked
• Sample size justified (power analysis)
• Multiple comparisons corrected
• Effect sizes reported
• Confidence intervals included
• Results correctly interpreted
• Limitations acknowledged

References

See references/ folder for:

• test_selection.md: Detailed test selection guide
• apa_reporting.md: Complete APA reporting templates

Converted and distributed by TomeVault — claim your Tome and manage your conversions.