Data Science

Python Data Science

NumPy and pandas best practices for data science and ML code. Designed for AI agents and LLMs to write fast, correct, and reproducible numeric code.

Numeric code has idioms that differ from general Python: loops are the anti-pattern, data structures are immutable buffers, and dtypes are a design decision. Where a rule here overlaps a coding-standards rule, this skill takes precedence for array and DataFrame code.

Categories

Vectorization [CRITICAL]

Keep computation in compiled NumPy/pandas code instead of the Python interpreter.

Indexing & Mutation [CRITICAL]

Write to DataFrames in ways that are correct under Copy-on-Write and visible in the data flow.

Data Types [HIGH]

Make dtypes an explicit decision rather than an inference result.

Schema Validation [HIGH]

Validate tabular data where it enters the system.

Reproducibility [HIGH]

Make experiments repeatable and free of hidden global state.

Transformation Style [MEDIUM]

Write DataFrame transformations that read top-to-bottom as a single pipeline.

Typing [MEDIUM]

Type hints that carry dtype and column information.

Testing [MEDIUM]

Assertions that fit floating-point arrays and DataFrames.

Quick Reference

Vectorization

# Whole-array operations, not loops / iterrows / apply(axis=1)
df["total"] = df["price"] * df["qty"]
rate = np.select([df["qty"] >= 100, df["qty"] >= 10], [0.2, 0.1], default=0.0)

# Collect, then build once
records = [{"id": item.id, "score": score(item)} for item in items]
results = pd.DataFrame.from_records(records)
combined = pd.concat(frames, ignore_index=True)

Indexing & Mutation

# .loc in one step; chained assignment never reaches df under Copy-on-Write
df.loc[df["status"] == "pending", "priority"] = "high"

# Returning form, not inplace=True
df = df.dropna(subset=["email"]).sort_values("created_at")

Data Types

df = pd.read_csv(
    path,
    dtype={"customer_id": "Int64", "zip_code": "string", "region": "category"},
    parse_dates=["created_at"],
)
SEVERITY = pd.CategoricalDtype(["low", "medium", "high"], ordered=True)
embeddings = np.zeros((n, 768), dtype=np.float32)

Schema Validation

import pandera.pandas as pa
from pandera.typing import DataFrame, Series


class OrderSchema(pa.DataFrameModel):
    order_id: Series[int] = pa.Field(unique=True)
    price: Series[float] = pa.Field(ge=0)

    class Config:
        strict = True


def load_orders(path: str) -> DataFrame[OrderSchema]:
    return OrderSchema.validate(pd.read_csv(path))

Reproducibility

def run_experiment(seed: int) -> Result:
    rng = np.random.default_rng(seed)
    split_rng, noise_rng = rng.spawn(2)
    ...

def add_noise(x: np.ndarray, rng: np.random.Generator) -> np.ndarray:
    return x + rng.normal(scale=0.1, size=x.shape)

Transformation Style

monthly = (
    pd.read_csv("sales.csv", parse_dates=["date"])
    .assign(
        revenue=lambda d: d["price"] * d["qty"],
        month=lambda d: d["date"].dt.to_period("M"),
    )
    .loc[lambda d: d["revenue"] > 0]
    .groupby("month", as_index=False)
    .agg(total_revenue=("revenue", "sum"), orders=("order_id", "count"))
)

Typing

import numpy.typing as npt

def to_float32(values: npt.ArrayLike) -> npt.NDArray[np.float32]:
    return np.asarray(values, dtype=np.float32)

Testing

np.testing.assert_allclose(result, expected, rtol=1e-6, atol=1e-12)
pd.testing.assert_frame_equal(result_df, expected_df, check_like=True)

See Also

• coding-standards - General Python rules; validation-pydantic, design-no-global-singleton, and doc-type-hints have direct counterparts here
• testing - pytest structure, fixtures, and mocking that numeric tests still follow