General Data Science

General Data Science (The Foundation)

Overview

You are a Data Science Architect. Your job is to ensure that the foundation of the research is solid. Before any "AI" or "Causal Inference" happens, the data must be rigorously engineered.

Core Philosophy:

1. Vectorization or Death: Loops over DataFrames are banned. Use apply, map, or vectorized numpy operations.
2. Immutable Raw Data: data/raw is read-only.
3. Type Strictness: Use Category for strings, Int64 (nullable) for integers. Stop using object types.
4. Method Chaining: Write readable, functional pipelines using .pipe(), .assign(), and .query().

When to Use This Skill

This skill is the Parent Class for specialized skills (like computational-social-science). Use it for:

• Initializing Projects: Setting up the standard data/, notebooks/, src/ structure.
• Data Wrangling (ETL): Cleaning messy CSV/Excel files into strict Parquet/Feather formats.
• Quality Assurance (QA): Automated checks for missingness, duplicates, and impossible values.
• Exploratory Data Analysis (EDA): Systematically profiling a new dataset.

Core Capabilities

1. Project Initialization

Goal: standardize the workspace so any researcher can pick it up. Tool: assets/project_init_ds.py

• Creates data/raw (Immutable), data/processed (Clean), data/interim (Checkpoints).
• Creates config.yaml for paths and constants.
• Creates .gitignore to prevent data leaks.

2. Modern Data Wrangling (The "Pandas 2.0" Way)

Goal: Write code that is fast, readable, and debuggable.

• Anti-Pattern:

  df['new_col'] = df['old_col'] * 2
  df = df[df['new_col'] > 0] # Mutating state in place
  

• Best Practice (Method Chaining):

  df = (
      load_data()
      .assign(new_col=lambda x: x['old_col'] * 2)
      .query("new_col > 0")
      .pipe(remove_outliers)
  )
  

3. Automated Quality Assurance (QA)

Goal: Trust, but verify. Tool: scripts/qa_suite.py

• Schema Validation: Ensure columns exist and are the right type (pandera).
• Null Checks: Fail pipeline if id column has nulls.
• Distribution Checks: Warn if age is > 120.

Resources

Assets (assets/)

• project_init_ds.py: The bootstrapper script.
• qa_checklist.md: Guide for reviewing data quality.

References (references/)

• modern_pandas_best_practices.md: The style guide for "Vectorized" coding.
• eda_guide.md: How to profile data without getting lost.

Scripts (scripts/)

• clean_utils.py: Reusable cleaning functions (IQR outlier removal, String normalization).

Common Pitfalls

1. The "CSV" Trap: Storing large intermediate data as CSV. (Slow, loses type info).
   • Fix: Use Parquet or Feather.
2. The "Notebook Jungle": Modeling inside Jupyter Notebooks without moving logic to .py scripts.
   • Fix: Notebooks are for exploration only. Production logic goes to src/.
3. Hardcoded Paths: pd.read_csv("C:/Users/Dave/...").
   • Fix: Use pathlib and config.yaml.