Feature Engineering

Feature Engineering (Verbose)

Core Patterns

Categorical Encoding

Cardinality and model family jointly determine the encoding. There is no default that survives both a 3-level column and a 200,000-level user id.

from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, TargetEncoder

OneHotEncoder(handle_unknown="ignore", min_frequency=0.01)   # rare levels pooled
OrdinalEncoder(categories=[["low", "medium", "high"]])       # order stated explicitly
TargetEncoder(smooth="auto", cv=5)                           # cross-fitted internally

min_frequency folds long-tail levels into an infrequent bucket, which both bounds the column count and prevents a level seen twice in training from acquiring its own weight.

Target Encoding Without Leakage

Naive target encoding is the most common self-inflicted leak in tabular work:

# ❌ each row's own label feeds its own feature value
df["city_te"] = df.groupby("city")["target"].transform("mean")

For a city appearing once, that feature is the label. Cross-validation scores climb, production performance does not. The fix is out-of-fold computation plus shrinkage:

from sklearn.preprocessing import TargetEncoder
from sklearn.pipeline import Pipeline

model = Pipeline([
    ("te", TargetEncoder(target_type="binary", smooth="auto", cv=5)),
    ("clf", HistGradientBoostingClassifier()),
])

Smoothing blends the category mean toward the global mean in proportion to the category's sample count: a level with n=2 lands near the prior, a level with n=5,000 near its own mean. Without it, rare categories become high-variance near-perfect predictors on the training set alone.

Numeric Transformation

from sklearn.preprocessing import (StandardScaler, RobustScaler,
                                   QuantileTransformer, KBinsDiscretizer)
import numpy as np

StandardScaler()                                   # (x - μ) / σ; assumes roughly symmetric
RobustScaler()                                     # median/IQR; outlier-tolerant
QuantileTransformer(output_distribution="normal")  # forces normality, rank-preserving
np.log1p(x)                                        # right-skewed counts, prices
KBinsDiscretizer(n_bins=10, encode="ordinal", strategy="quantile")

Which models need it:

log1p is the right choice over log whenever zeros occur, which for counts and monetary amounts is nearly always.

Temporal Features

ts = pd.to_datetime(df["event_time"])

df["hour"]        = ts.dt.hour
df["dayofweek"]   = ts.dt.dayofweek
df["is_weekend"]  = (ts.dt.dayofweek >= 5).astype(int)
df["days_since"]  = (reference_time - ts).dt.days

# cyclical: 23:00 and 00:00 should be neighbours
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

Both sine and cosine are required — either alone maps two distinct times to the same value. And days_since must be measured from a per-row reference (the prediction timestamp), not from datetime.now(), or the feature drifts every time you retrain and means something different in backtest than in production.

Entity Aggregates With Point-in-Time Correctness

Aggregates over an entity's history usually carry more signal than any raw field:

agg = (events[events.ts < cutoff]
       .groupby("user_id")
       .agg(txn_count=("amount", "size"),
            txn_mean=("amount", "mean"),
            txn_max=("amount", "max"),
            last_seen=("ts", "max")))
agg["recency_days"] = (cutoff - agg["last_seen"]).dt.days

The ts < cutoff filter is the whole game. Aggregating a user's full transaction history to predict a fraud event that occurred midway through it means the mean already includes the fraudulent transaction. The model learns to detect its own label. This class of bug produces excellent offline metrics and a model that is useless live — see cross-validation-strategies for the splitting side of the same discipline.

Interactions and Ratios

Trees approximate interactions through repeated splits but represent ratios poorly. Stating them explicitly often helps more than adding rows:

df["price_per_sqft"]   = df["price"] / df["sqft"]
df["debt_to_income"]   = df["debt"] / df["income"].replace(0, np.nan)
df["util_ratio"]       = df["balance"] / df["credit_limit"]

Guard denominators. A silent inf propagates into the model and, for linear models, into an immediate failure to converge.

Feature Selection

from sklearn.feature_selection import VarianceThreshold, SelectKBest, mutual_info_classif
from sklearn.inspection import permutation_importance

VarianceThreshold(0.0)                      # drop constants
SelectKBest(mutual_info_classif, k=50)      # captures non-linear dependence
permutation_importance(model, X_val, y_val, n_repeats=10)

Prefer permutation importance on a validation set over a tree's built-in feature_importances_. The impurity-based version is biased toward high-cardinality and continuous features — a random unique id can outrank a genuinely predictive binary flag, purely because it offers more split points.

Common Anti-Patterns

❌ df.groupby(cat)["target"].transform("mean") as a feature. ✅ TargetEncoder with cross-fitting and smoothing, inside the pipeline.

❌ Fitting scalers or imputers on the full dataset before splitting. ✅ Every stateful transform inside a Pipeline, refit per fold.

❌ Aggregating an entity's entire history regardless of the label timestamp. ✅ Point-in-time filters on every aggregate window.

❌ One-hot encoding a high-cardinality id — thousands of near-empty columns. ✅ Target/count encoding, hashing, or a model with native categorical support.

❌ Feeding raw epoch timestamps to a tree. ✅ Decompose into hour/dow/month plus cyclical pairs and elapsed-time features.

❌ Dropping rows with nulls, or imputing without an indicator. ✅ SimpleImputer(add_indicator=True) — missingness is frequently predictive.

❌ Trusting feature_importances_ for feature choice. ✅ Permutation importance on held-out data.

❌ Engineering a feature that cannot be computed at serve time — a field populated by a batch job that lands hours later. ✅ Check availability and latency of every input before building on it.

Feature Engineering Checklist

• Every stateful transform lives inside a Pipeline
• handle_unknown="ignore" set on one-hot encoders
• Target encoding cross-fitted and smoothed
• Encoding chosen per column by cardinality
• Scaling applied where the model family requires it
• Skewed numerics log- or quantile-transformed
• Timestamps decomposed; cyclical features use sin and cos
• Every entity aggregate has a point-in-time cutoff
• Missingness indicators retained
• Denominators guarded against zero
• Suspiciously dominant features investigated for leakage
• Feature importance measured by permutation on validation data
• Every feature reproducible from data available at prediction time
• Transformation logic shared between training and serving, not reimplemented