paulpas Linear Regression
Comprehensive guide to linear regression in machine learning and data science workflows.
When to Use This Skill
- Solving real-world supervised learning problems
- Building machine learning pipelines with linear regression
- Implementing best practices for linear regression
- Optimizing model performance using linear regression techniques
- Learning industry-standard approaches to linear regression
When NOT to Use This Skill
- When using pre-built libraries without understanding underlying concepts
- For toy problems that don't require linear regression rigor
- When domain expertise in specific problem requires different approach
- If your problem doesn't require the complexity this skill provides
Purpose and Key Concepts
Linear Regression is a critical component of the machine learning workflow. This skill covers:
- Theoretical foundations — Mathematical principles and statistical concepts
- Practical implementation — Working code examples and patterns
- Common pitfalls — Mistakes to avoid and how to recover from them
- Best practices — Industry-standard approaches and optimization techniques
Core Workflow
- Understand the problem — Clearly define what you're solving for
- Select approach — Choose the right technique for your data and constraints
- Implement solution — Write clean, tested code following best practices
- Validate results — Verify your implementation with tests and validation
- Optimize performance — Improve efficiency and accuracy incrementally
Implementation Patterns
Pattern 1: Basic Linear Regression
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
# Generate synthetic regression data for demonstration
X, y = np.random.randn(500, 5), np.random.randn(500)
df = pd.DataFrame(X, columns=[f'feature_{i}' for i in range(5)])
df['target'] = y
# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
df.drop('target', axis=1), df['target'], test_size=0.2, random_state=42
)
# Initialize and train the OLS model
model = LinearRegression()
model.fit(X_train, y_train)
# Generate predictions and compute evaluation metrics
y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f"Mean Squared Error: {mse:.4f}")
print(f"R² Score: {r2:.4f}")
print(f"Coefficients: {model.coef_}")
print(f"Intercept: {model.intercept_:.4f}")
Pattern 2: Production-Ready Linear Regression
import logging
import pandas as pd
import numpy as np
from typing import Any, Dict
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.preprocessing import StandardScaler
logger = logging.getLogger(__name__)
class LinearRegressionPipeline:
"""Production-ready linear regression with validation and metrics."""
def __init__(self, random_state: int = 42) -> None:
self.random_state = random_state
self.model = LinearRegression()
self.scaler = StandardScaler()
self.is_fitted: bool = False
def _validate_input(self, data: pd.DataFrame) -> pd.DataFrame:
"""Validate input DataFrame for numeric compatibility."""
if data.empty:
raise ValueError("Input DataFrame cannot be empty.")
numeric_cols = data.select_dtypes(include='number')
if numeric_cols.shape[1] == 0:
raise ValueError("All columns must be numeric for linear regression.")
return data
def execute(self, data: pd.DataFrame, target_col: str = 'target') -> Dict[str, Any]:
"""Execute linear regression pipeline on provided data."""
try:
df = self._validate_input(data)
X = df.drop(columns=[target_col])
y = df[target_col]
# Scale features for numerical stability and convergence
X_scaled = self.scaler.fit_transform(X)
# Train model on scaled data
self.model.fit(X_scaled, y)
self.is_fitted = True
# Predict and evaluate performance
y_pred = self.model.predict(X_scaled)
mae = mean_absolute_error(y, y_pred)
r2 = r2_score(y, y_pred)
return {
'status': 'success'
'mae': float(mae)
'r2_score': float(r2)
'coefficients': self.model.coef_.tolist()
'intercept': float(self.model.intercept_)
'feature_names': X.columns.tolist()
}
except Exception as e:
logger.error(f"Regression pipeline failed: {e}")
return {'status': 'error', 'message': str(e)}
BAD vs GOOD Implementation
# BAD: Hardcoded values, no validation, bypasses error handling, violates DRY
def bad_regression(df):
X = df.iloc[:, :5]
y = df.iloc[:, 5]
model = LinearRegression()
model.fit(X, y)
return model.predict(X)
# GOOD: Type hints, validation, modular design, follows SOLID/DRY principles
def good_regression(df: pd.DataFrame, target: str = 'target') -> Dict[str, Any]:
if df.empty:
raise ValueError("DataFrame cannot be empty")
X = df.drop(columns=[target])
y = df[target]
model = LinearRegression()
model.fit(X, y)
return {'predictions': model.predict(X), 'coefficients': model.coef_}
Best Practices
- ✅ Always validate your implementation on test data
- ✅ Document your assumptions and methodology
- ✅ Use version control for reproducibility
- ✅ Monitor performance metrics in production
- ✅ Periodically review and update your approach
- ✅ Test with edge cases and outliers
- ✅ Log all significant operations for debugging
Common Pitfalls
| Pitfall | Problem | Solution | |
Constraints
MUST DO
- Validate linearity assumption using residual plots before interpreting coefficients
- Check for multicollinearity using VIF (Variance Inflation Factor) — flag any predictor with VIF > 10
- Standardize or normalize features before regularization (Ridge, Lasso, ElasticNet)
- Report both R-squared and adjusted R-squared, and include confidence intervals for all coefficients
MUST NOT DO
- Do not interpret correlation as causation from regression output without controlled experiments
- Avoid including dummy variables for all categories without dropping one reference category (dummy variable trap)
- Never use OLS on time series data without checking for stationarity — leads to spurious regression
- Do not report R-squared alone; always include residual diagnostics and cross-validated error metrics
Live References
Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content.