Data Science

Data Science

Intro

A solid data analysis flows from import through cleaning, tidying, exploration, modeling, and communication — in that order. Tidy data, honest uncertainty, and plain-language findings beat clever models on messy inputs.

Overview

Import and cleaning

Inspect shape, dtypes, nulls, and duplicates immediately after loading. Print df.info() and df.describe() before any analysis. Handle missing data explicitly — drop, fill, or flag, but never ignore silently. Parse dates at load time (parse_dates=) and set correct dtypes early. Validate assumptions: no negatives where impossible, no string/numeric mismatches.

Tidy data

Reshape to tidy form: one variable per column, one observation per row. Use melt() for wide-to-long, pivot_table() for long-to-wide. Split mixed columns (e.g. city_state into two). Normalize repeated entities into separate tables. See Level 3 for messy-pattern recipes.

Exploratory data analysis

Start univariate (histograms, value counts, box plots), then bivariate (scatter, grouped bars, correlation). Look for outliers, skewness, class imbalance, and unexpected patterns. Use .groupby() to understand subgroups. Summarize findings in plain language before moving to modeling.

Statistical reasoning

State the question before choosing a test. Check assumptions (normality, equal variance, independence) before applying parametric tests. Report effect sizes alongside p-values — statistical significance is not practical significance. Use confidence intervals to communicate uncertainty.

Feature selection

Remove zero- and near-zero-variance features. Drop one of each highly correlated pair (r > 0.9). Use domain knowledge first, then data-driven methods (mutual information, permutation importance). Watch for leakage — never use target-derived features in training.

Model selection workflow

Start simple: a mean/median baseline, then linear or logistic regression. Add complexity only if simpler models underperform and you understand why. Use train/validation/test split or cross-validation; never evaluate on training data. Compare on a single metric chosen before fitting. Document why a model was chosen, not just which one won.

Communicating results

Lead with the finding, not the method. Use plain language: "customers who used feature X retained 23% more" beats "the coefficient was 0.23." Show uncertainty — ranges, intervals, caveats. Include a "so what" — what action does this analysis support? Pin library versions, set random seeds, and script everything for reproducibility.

Preferred libraries

Gotchas

Agent-specific failure modes — provider-neutral pause-and-self-check items:

• Starting with a complex model before a simple baseline. Training an XGBoost model before computing a mean predictor or running a logistic regression means you can't know how much of the model's value came from complexity vs. data. Build a baseline first; complexity is only justified when the baseline is inadequate.
• Evaluating on training data. "The model is 98% accurate" on the data it trained on is not a finding — it is a description of memorization. Always evaluate on held-out data and report the held-out metric.
• Confusing statistical significance with practical significance. With n > 10,000, almost any real difference is statistically significant. A p-value of 0.001 on a 0.1% conversion difference means "this is real" but does not mean "this matters." Always report effect size alongside p-values.
• Choosing a statistical test before checking its assumptions. Running a t-test on non-normally distributed data with small samples produces unreliable p-values. Check normality (Shapiro-Wilk for n < 5000), equal variance (Levene), and independence before choosing a parametric test; fall back to non-parametric alternatives when assumptions fail.
• Reporting findings without uncertainty. "Feature X predicts churn" without a confidence interval or cross-validation variance means the finding might not replicate. Always communicate uncertainty ranges alongside point estimates.
• Truncating the bar chart y-axis to make differences look larger. A bar chart with a y-axis starting at 0.85 makes a 0.03 difference look dramatic. Unless the baseline is well-understood (like conversion rate from 30% to 32%), bar charts should start at zero.
• Not setting random seeds before any analysis. Non-reproducible analysis is not science. Set random_state=42 (or equivalent) on all sampling, model initialization, and cross-validation calls. Document the seed in the notebook or script so results can be reproduced exactly.

Full reference

Tidy data — messy patterns and fixes

Headers as values (years across columns). Melt to long form:

# pandas
df_tidy = df.melt(id_vars="country", var_name="year", value_name="population")

# polars
df_tidy = df.unpivot(index="country", variable_name="year", value_name="population")

Multiple variables in one column (male_18-25). Split:

# pandas
df[["gender", "age_group"]] = df["demographic"].str.split("_", expand=True)

Variables in both rows and columns (metric in rows, dates in columns). Melt dates first, then pivot metrics:

long = df.melt(id_vars=["id", "metric"], var_name="month", value_name="value")
tidy = long.pivot_table(index=["id", "month"], columns="metric", values="value").reset_index()

Multiple observational units in one table. Normalize:

songs = df[["song_id", "title", "artist"]].drop_duplicates()
plays = df[["song_id", "play_date", "user_id"]]

Wide form is fine when feeding correlation matrices, display pivot tables, or ML feature matrices. Store and clean tidy; reshape for the consumer.

Statistical methods

Always compute descriptive stats first (df.describe(), value_counts(normalize=True), df.skew()). Mean is sensitive to outliers; median is robust. Standard deviation reports spread in original units; IQR is the robust analogue.

Choosing a test:

Check normality with shapiro (n < 5000), equal variance with levene, and chi-square cell counts >= 5 (otherwise Fisher's exact).

A p-value is the probability of data this extreme if H0 is true — not the probability that H0 is true. A 95% CI means 95% of intervals from repeated studies contain the true value. p < 0.05 with a tiny effect is significant but irrelevant; p = 0.06 is not "trending." For multiple comparisons use Bonferroni or FDR correction.

For regression, use statsmodels OLS for continuous outcomes and check residual normality, homoscedasticity, and VIF < 5 for multicollinearity. Use logistic regression for binary outcomes; exponentiate coefficients for odds ratios.

Practical guidance: small samples (n < 30) call for non-parametric tests; large samples (n > 10000) make almost everything significant, so focus on effect size. Understand missingness mechanism before imputing. Bayesian methods (PyMC, ArviZ) when you actually need P(hypothesis | data).

Visualization guidelines

Default rule: bar (categorical) or scatter (continuous) when unsure.

Basic seaborn setup uses sns.set_theme(style="whitegrid", palette="colorblind"). Categorical palettes: "colorblind" or px.colors.qualitative.Safe. Sequential: viridis, cividis. Diverging: coolwarm, RdBu. Avoid rainbow/jet and red-green-only distinctions. Never rely on color alone — add patterns, markers, or labels. Test with a colorblindness simulator.

Annotate the insight, not just the data: peak labels, threshold lines, events that explain trend changes. Use direct labels instead of legends with 2–3 series. Plotly for interactive (notebooks, dashboards); matplotlib/seaborn for static (reports, papers).

Export checklist: title states the finding, both axes labeled with units, legend or direct labels, colorblind-friendly palette, 300 dpi for print and SVG/PDF for scalable, fonts readable at final size.

Common mistakes: truncated y-axis on bar charts, dual y-axes, 3D charts, pies with > 4 slices, missing units, default titles like "Figure 1," and overplotting (use alpha, hexbin, or 2D density).

Worked examples

• CSV of transactions, churn patterns: Load, inspect shape/dtypes/nulls, build a tidy customer time series, run EDA on churn-rate distributions and cohorts, t-test usage frequency between churned and retained groups (with effect size), produce annotated visualizations of key drivers.
• Top features for house prices: Load, clean nulls, encode categoricals, check correlations, fit a baseline linear regression, use permutation importance to rank features, report top 10 with CIs and a bar chart, note multicollinearity concerns.
• A/B conversion difference significant? Compute rates and sample sizes, run chi-square (verify expected cell counts >= 5), report p-value and 95% CI for the difference, visualize rates with error bars, state whether the effect is practically meaningful.