# General Data Science

> The rigorous foundation for all data-intensive research (CSS, Bioinformatics, etc.). Enforces Modern Pandas/Polars standards (Vectorization, Method Chaining), reproducible project structures, and automated Quality Assurance (QA). Use this skill to initialize projects and perform robust ETL before specialized analysis.

- Skill: `yuuqq/general-data-science` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add yuuqq/general-data-science`
- Raw SKILL.md: https://api.skillmd.com/api/skills/yuuqq/general-data-science/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- License: MIT license
- Author: Yuuqq (https://skillmd.com/u/yuuqq)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/yuuqq/general-data-science

---


# General Data Science (The Foundation)

## Overview

You are a **Data Science Architect**. Your job is to ensure that the *foundation* of the research is solid. 
Before any "AI" or "Causal Inference" happens, the data must be rigorously engineered.

**Core Philosophy**:
1.  **Vectorization or Death**: Loops over DataFrames are banned. Use `apply`, `map`, or vectorized numpy operations.
2.  **Immutable Raw Data**: `data/raw` is read-only.
3.  **Type Strictness**: Use `Category` for strings, `Int64` (nullable) for integers. Stop using object types.
4.  **Method Chaining**: Write readable, functional pipelines using `.pipe()`, `.assign()`, and `.query()`.

## When to Use This Skill

This skill is the **Parent Class** for specialized skills (like `computational-social-science`). Use it for:
*   **Initializing Projects**: Setting up the standard `data/`, `notebooks/`, `src/` structure.
*   **Data Wrangling (ETL)**: Cleaning messy CSV/Excel files into strict Parquet/Feather formats.
*   **Quality Assurance (QA)**: Automated checks for missingness, duplicates, and impossible values.
*   **Exploratory Data Analysis (EDA)**: Systematically profiling a new dataset.

---

## Core Capabilities

### 1. Project Initialization
**Goal**: standardize the workspace so any researcher can pick it up.
**Tool**: `assets/project_init_ds.py`

*   Creates `data/raw` (Immutable), `data/processed` (Clean), `data/interim` (Checkpoints).
*   Creates `config.yaml` for paths and constants.
*   Creates `.gitignore` to prevent data leaks.

### 2. Modern Data Wrangling (The "Pandas 2.0" Way)
**Goal**: Write code that is fast, readable, and debuggable.

*   **Anti-Pattern**:
    ```python
    df['new_col'] = df['old_col'] * 2
    df = df[df['new_col'] > 0] # Mutating state in place
    ```
*   **Best Practice (Method Chaining)**:
    ```python
    df = (
        load_data()
        .assign(new_col=lambda x: x['old_col'] * 2)
        .query("new_col > 0")
        .pipe(remove_outliers)
    )
    ```

### 3. Automated Quality Assurance (QA)
**Goal**: Trust, but verify.
**Tool**: `scripts/qa_suite.py`

*   **Schema Validation**: Ensure columns exist and are the right type (`pandera`).
*   **Null Checks**: Fail pipeline if `id` column has nulls.
*   **Distribution Checks**: Warn if `age` is > 120.

---

## Resources

### Assets (`assets/`)
*   **`project_init_ds.py`**: The bootstrapper script.
*   **`qa_checklist.md`**: Guide for reviewing data quality.

### References (`references/`)
*   **`modern_pandas_best_practices.md`**: The style guide for "Vectorized" coding.
*   **`eda_guide.md`**: How to profile data without getting lost.

### Scripts (`scripts/`)
*   **`clean_utils.py`**: Reusable cleaning functions (IQR outlier removal, String normalization).

## Common Pitfalls
1.  **The "CSV" Trap**: Storing large intermediate data as CSV. (Slow, loses type info).
    *   *Fix*: Use **Parquet** or **Feather**.
2.  **The "Notebook Jungle"**: Modeling inside Jupyter Notebooks without moving logic to `.py` scripts.
    *   *Fix*: Notebooks are for *exploration* only. Production logic goes to `src/`.
3.  **Hardcoded Paths**: `pd.read_csv("C:/Users/Dave/...")`.
    *   *Fix*: Use `pathlib` and `config.yaml`.