Dimensionality Reduction

Dimensionality Reduction (Verbose)

Core Patterns

Deciding Whether to Reduce at All

Reduction is a trade, not an improvement. You give up interpretability and some signal in exchange for speed, decorrelation, or a picture. Reach for it when at least one of these holds:

• Features vastly outnumber samples (p ≫ n) and a linear model cannot be regularized enough
• Features are strongly collinear and the model is sensitive to that (linear, k-NN, k-means)
• Inference latency or memory is bounded and the feature vector dominates it
• You need a 2-D view of structure for human consumption

Gradient boosting on 200 moderately correlated tabular features needs none of this. Feeding it PCA components typically costs accuracy — the trees lose the axis-aligned splits that matched real variables — and buys nothing.

PCA

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("pca", PCA(n_components=0.95, svd_solver="full", random_state=0)),
    ("clf", LogisticRegression(max_iter=1000)),
])

PCA finds orthogonal directions of maximum variance via the SVD of the centred data matrix. Two properties follow directly and explain most misuse:

It is scale-dependent. Variance is measured in the units of each column. A feature in dollars will dominate one in years by a factor of millions. PCA centres but never standardizes, so StandardScaler is mandatory for mixed units.

It is unsupervised. The directions of greatest variance need not be the directions that separate classes. A low-variance component can carry all the discriminative signal, and dropping it destroys the model while retaining 99% of variance.

Choosing the rank:

pca = PCA().fit(X_scaled)
cum = pca.explained_variance_ratio_.cumsum()
print(f"90%: {(cum < 0.90).sum() + 1} comps, 95%: {(cum < 0.95).sum() + 1} comps")

Passing a float to n_components selects the smallest rank reaching that ratio. Better still, treat it as a hyperparameter and let the downstream metric decide:

GridSearchCV(pipe, {"pca__n_components": [10, 25, 50, 100]}, cv=5, scoring="roc_auc")

For data too large for memory, IncrementalPCA fits in partial_fit batches at a small cost in precision.

Sparse Data

from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer

X = TfidfVectorizer(max_features=50_000).fit_transform(docs)   # sparse
svd = TruncatedSVD(n_components=300, random_state=0)           # LSA
Z = svd.fit_transform(X)

Never call PCA on a sparse matrix. Centring destroys sparsity: a 50,000-column TF-IDF matrix at 0.1% density becomes fully dense and blows past available memory. TruncatedSVD skips the centring step and operates on the sparse matrix directly. Because it does not centre, its components are not identical to PCA's even on the same data — a difference that matters when reproducing published numbers.

Supervised Reduction

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

lda = LDA(n_components=2)      # capped at n_classes - 1
Z = lda.fit_transform(X_scaled, y)

LDA maximizes between-class over within-class scatter, so it targets separability rather than variance. The hard cap at n_classes - 1 components is structural: binary classification yields exactly one dimension. Because it uses y, LDA must be fitted inside the CV fold like any supervised step.

Manifold Methods for Visualization

from sklearn.manifold import TSNE

Z = TSNE(n_components=2, perplexity=30, init="pca", random_state=0).fit_transform(X)

t-SNE preserves local neighbourhoods and deliberately discards global geometry. Three consequences are routinely misread:

1. Distances between clusters are meaningless. Two clusters at opposite corners may be adjacent in the source space.
2. Cluster sizes are meaningless. t-SNE equalizes density; a large blob is not a larger or more variable group.
3. perplexity changes the answer. At 5 the plot fragments into micro-clusters; at 50 it merges them. Run several values before drawing a conclusion.

t-SNE exposes no transform, so it cannot embed unseen points and cannot appear in a serving pipeline. UMAP can (umap.UMAP().transform), is much faster at scale, and retains more global structure — but it is still a visualization tool first, and any claim that its clusters are "real" needs corroboration from the original space.

Standard practice on wide data: PCA down to ~50 components first, then t-SNE or UMAP. This denoises and cuts runtime substantially.

Feature Selection as the Alternative

When interpretability, auditability, or serving cost matter, selecting a subset beats projecting into a new basis — you keep column names and can stop computing the columns you dropped.

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LassoCV

sel = SelectFromModel(LassoCV(cv=5)).fit(X_scaled, y)   # L1 drives weights to zero
kept = X.columns[sel.get_support()]

PCA still requires all original inputs at inference time — every component is a combination of all of them — so it reduces model width but not data-collection cost. Selection reduces both.

Common Anti-Patterns

❌ PCA on unstandardized features of mixed units. ✅ StandardScaler immediately before PCA, in the same pipeline.

❌ PCA on a sparse matrix — centring densifies and exhausts memory. ✅ TruncatedSVD.

❌ Fitting the reducer on the full dataset before splitting — validation covariance leaks into the basis. ✅ Reducer as a Pipeline step, refit per fold.

❌ Interpreting PC1 as a business variable. ✅ Treat components as opaque; use feature selection when explanation is required.

❌ Reading inter-cluster distances or cluster sizes off a t-SNE plot. ✅ Read only local neighbourhood membership; verify in the original space.

❌ t-SNE embeddings as model features — no transform, non-deterministic, not serveable. ✅ PCA or UMAP if an embedding must be applied to new data.

❌ Reducing before a gradient-boosted tree to "help" it. ✅ Trees handle wide correlated inputs; measure before assuming reduction helps.

❌ Fixing n_components=2 because it plots nicely, then modelling on it. ✅ Tune the rank against the downstream metric.

❌ Assuming 95% retained variance means 95% retained signal. ✅ Compare end-to-end scores with and without reduction.

Dimensionality Reduction Checklist

• Reduction justified by p ≫ n, collinearity, latency, or visualization
• Features standardized before PCA
• TruncatedSVD used for sparse input, not PCA
• Reducer inside a Pipeline, fitted per fold
• n_components chosen from the variance curve or tuned by CV
• explained_variance_ratio_ inspected, not assumed
• Baseline model without reduction measured for comparison
• Components never presented as interpretable variables
• perplexity varied before trusting any t-SNE structure
• t-SNE confined to visualization, never to a serving path
• PCA applied before t-SNE/UMAP on wide data
• Feature selection considered where interpretability or input cost matters
• Fitted reducer persisted with the model so serving uses the identical basis