R Feature Engineering

R Feature Engineering - Strategic Decisions and Selection

Expert guidance for making strategic decisions about feature engineering and selection in R, based on "Feature Engineering and Selection: A Practical Approach for Predictive Models" by Max Kuhn and Kjell Johnson.

Overview

This skill provides strategic guidance for feature engineering decisions - helping you understand when and why to use specific techniques, not just how to implement them. While r-tidymodels focuses on workflow execution, this skill helps you make informed decisions about:

• Encoding strategies - Choosing the right categorical encoding method
• Transformation approaches - Selecting appropriate numeric transformations
• Interaction detection - Systematically finding interaction effects
• Feature selection - Comparing filter, wrapper, and embedded methods
• Missing data strategies - Choosing imputation approaches

When This Skill Activates

Use this skill when you need to:

• Compare approaches: "Should I use dummy encoding or likelihood encoding?"
• Make strategic decisions: "What's the best feature selection method for my problem?"
• Understand trade-offs: "When should I use Box-Cox vs Yeo-Johnson?"
• Detect patterns: "How do I systematically find interactions?"
• Handle complexity: "How to deal with high-cardinality categorical variables?"

NOT for:

• Basic recipe implementation → Use r-tidymodels
• Data wrangling → Use r-datascience or tidyverse-expert
• General ML workflow → Use r-tidymodels

Core Philosophy

1. Re-representation First - Better features often beat better models
2. Domain Knowledge Matters - Statistical methods guide, but domain expertise decides
3. Validate Everything - All feature engineering must happen inside resampling
4. Interpretability vs Performance - Understand the trade-off for your use case
5. Prevent Data Leakage - Derive all parameters from training data only

Categorical Encoding Strategies

Decision Framework: Which Encoding to Use?

# Decision tree for categorical encoding
choose_encoding <- function(n_levels, n_obs, has_outcome, ordinal) {
  if (ordinal) return("step_ordinalscore()")

  if (n_levels < 10) {
    return("step_dummy() # Simple and interpretable")
  }

  if (n_levels <= 50) {
    if (n_obs < 1000 && has_outcome) {
      return("embed::step_lencode_mixed() # Supervised encoding")
    } else {
      return("step_dummy() or step_other() |> step_dummy()")
    }
  }

  if (n_levels > 50) {
    if (n_obs < 1000 && has_outcome) {
      return("embed::step_embed() # Entity embeddings")
    } else {
      return("textrecipes::step_texthash() # Feature hashing")
    }
  }
}

Method Comparison

Detailed Encoding Approaches

See references/categorical-encoding.md for:

• Step-by-step implementation of each method
• When to use supervised vs unsupervised encoding
• Handling novel categories in production
• Preventing data leakage with proper CV
• Case studies and comparisons

Numeric Transformation Strategies

Transformation Decision Tree

# When to transform numeric predictors
needs_transformation <- function(predictor, model_type) {
  # Tree-based models don't need transformations
  if (model_type %in% c("random_forest", "xgboost", "decision_tree")) {
    return("No transformation needed - tree models are scale-invariant")
  }

  # Distance-based and linear models benefit
  skewness <- e1071::skewness(predictor)

  if (abs(skewness) < 0.5) {
    return("step_normalize() # Mild skewness, just standardize")
  }

  if (all(predictor > 0) && skewness > 1) {
    return("step_log() or step_BoxCox() # Right-skewed, positive values")
  }

  if (any(predictor <= 0) && abs(skewness) > 1) {
    return("step_YeoJohnson() # Skewed with zero/negative values")
  }

  return("step_normalize() # Default standardization")
}

Transformation Catalog

When NOT to Transform

# Models that DON'T benefit from transformations
no_transform_needed <- c(
  "random_forest",    # Split-based, scale-invariant
  "xgboost",          # Handles skewness naturally
  "decision_tree",    # Works on order only
  "naive_bayes"       # Uses distributions directly
)

# Models that REQUIRE transformations
transform_required <- c(
  "linear_reg",       # Assumes normality
  "logistic_reg",     # Benefits from standardization
  "svm_*",            # Distance-based, scale-sensitive
  "nearest_neighbor", # Euclidean distance
  "glmnet"            # Penalization scale-dependent
)

See references/numeric-transformations.md for:

• Detailed transformation techniques (1:1, 1:many, many:many)
• Basis expansions and splines
• Discretization strategies (use as last resort!)
• PCA and dimensionality reduction trade-offs

Detecting Interaction Effects

Why Interactions Matter

Interactions occur when the effect of one predictor depends on the value of another. Example:

• Effect of fertilizer depends on water availability
• Impact of age depends on education level
• Drug effectiveness depends on patient genetics

Systematic Detection Methods

# Method selection based on problem size
select_interaction_method <- function(n_predictors, n_obs) {
  if (n_predictors < 20) {
    return("Simple screening with nested models (exhaustive)")
  }

  if (n_predictors <= 50 && n_obs > 1000) {
    return("Random forest importance + interaction screening")
  }

  if (n_predictors > 50) {
    return("Two-stage: Lasso main effects → FSA for interactions")
  }

  if (n_obs / n_predictors < 10) {
    return("Penalized regression (glmnet) with all interactions")
  }
}

Four Detection Approaches

1. Simple Screening (Small p)

# Exhaustive search with proper CV
library(tidymodels)

# Test all pairwise interactions
interactions <- combn(predictors, 2, simplify = FALSE)

results <- map_dfr(interactions, function(pair) {
  formula <- as.formula(paste("outcome ~", paste(pair, collapse = " * ")))

  # Use resampling to avoid overfitting
  cv_results <- fit_resamples(
    workflow() |>
      add_formula(formula) |>
      add_model(linear_reg()),
    resamples = vfold_cv(train, v = 10)
  )

  collect_metrics(cv_results)
})

2. Random Forest Importance

# Use RF to identify important main effects, then test interactions
library(vip)

rf_fit <- rand_forest(trees = 1000) |>
  set_engine("ranger", importance = "permutation") |>
  fit(outcome ~ ., data = train)

# Get top predictors
top_vars <- vip(rf_fit, num_features = 10)$data$Variable

# Test interactions among top predictors only
interactions_to_test <- combn(top_vars, 2, simplify = FALSE)

3. Two-Stage Modeling

# Stage 1: Fit main effects model
main_effects <- linear_reg(penalty = 0.01, mixture = 1) |>
  set_engine("glmnet") |>
  fit(outcome ~ ., data = train)

# Stage 2: Look for interactions in residuals
train_with_resid <- train |>
  mutate(residual = outcome - predict(main_effects, train)$.pred)

# Now search for interactions that explain residuals
interaction_model <- fit(
  outcome ~ . + (age * income) + (education * experience),
  data = train_with_resid
)

4. Feasible Solution Algorithm (FSA)

# For very high-dimensional problems
# Iteratively substitute predictor combinations
# See references/interaction-detection.md for implementation

See references/interaction-detection.md for:

• Detailed implementation of each method
• When to use each approach
• Preventing overfitting in interaction search
• Real-world case studies

Feature Selection Strategies

Selection Method Taxonomy

Feature Selection
├── Filter Methods (Univariate)
│   ├── Correlation with outcome
│   ├── Statistical tests (t-test, chi-square)
│   └── Information gain / mutual information
├── Wrapper Methods (Model-based)
│   ├── Greedy Search
│   │   ├── Forward selection
│   │   ├── Backward elimination
│   │   └── Recursive Feature Elimination (RFE)
│   └── Global Search
│       ├── Genetic algorithms
│       ├── Simulated annealing
│       └── Particle swarm
└── Embedded Methods (During training)
    ├── Lasso (L1 penalty)
    ├── Elastic net
    └── Tree-based importance

Decision Framework

select_feature_method <- function(n_features, n_obs, need_optimal, compute_budget) {
  # Filter methods: Fast preprocessing
  if (n_features > 1000) {
    return("Filter methods first to reduce to <100 features")
  }

  # Embedded methods: Fast and effective
  if (compute_budget == "low" || n_features > 50) {
    return("Lasso or elastic net (embedded)")
  }

  # Wrapper methods: Best performance
  if (need_optimal && compute_budget == "high" && n_features < 50) {
    if (n_features < 20) {
      return("Recursive Feature Elimination (RFE)")
    } else {
      return("Genetic algorithm for global optimum")
    }
  }

  return("Hybrid: Filter → Lasso → RFE")
}

Method Comparison

Greedy Search Methods

# Forward Selection
library(caret)

ctrl <- rfeControl(
  functions = lmFuncs,  # or rfFuncs, nbFuncs
  method = "cv",
  number = 10
)

# Recursive Feature Elimination
rfe_results <- rfe(
  x = train[, predictors],
  y = train$outcome,
  sizes = c(5, 10, 15, 20, 25),
  rfeControl = ctrl
)

# Best subset
predictors(rfe_results)  # Selected features

Global Search Methods

# Genetic Algorithm for feature selection
library(GA)

# Fitness function: CV performance with selected features
fitness <- function(chromosome) {
  selected <- which(chromosome == 1)
  if (length(selected) == 0) return(0)

  # Fit model with selected features
  cv_results <- vfold_cv(train, v = 5) |>
    mutate(
      model = map(splits, ~{
        fit(linear_reg(), outcome ~ .,
            data = analysis(.x)[, c(selected, "outcome")])
      }),
      pred = map2(model, splits, ~predict(.x, assessment(.y))),
      truth = map(splits, ~assessment(.x)$outcome)
    )

  # Return negative RMSE (GA maximizes)
  -mean(map2_dbl(cv_results$pred, cv_results$truth,
                 ~sqrt(mean((.x$.pred - .y)^2))))
}

# Run GA
ga_result <- ga(
  type = "binary",
  fitness = fitness,
  nBits = ncol(train) - 1,  # One bit per predictor
  maxiter = 50,
  popSize = 20
)

See references/feature-selection.md for:

• Complete greedy search implementations
• Global search algorithms (SA, PSO)
• Hybrid strategies
• Computational considerations
• Case studies with performance comparisons

Missing Data Strategies

Is Missingness Informative?

# Test if missingness predicts outcome
library(naniar)

# Create missingness indicators
data_with_miss <- train |>
  bind_shadow() |>  # Adds *_NA columns
  mutate(across(ends_with("_NA"), ~if_else(.x == "!NA", 1, 0)))

# Test association
miss_importance <- rand_forest() |>
  fit(outcome ~ ., data = select(data_with_miss, outcome, ends_with("_NA"))) |>
  vip::vip()

# If missingness indicators are important → keep them!

Strategy Selection

# Decision tree for missing data
handle_missing <- function(var, mechanism, importance) {
  if (mechanism == "MCAR" && importance == "low") {
    return("step_impute_median() # Simple imputation")
  }

  if (mechanism == "MAR") {
    return("step_impute_knn() or step_impute_bag() # Model-based")
  }

  if (importance == "high" || mechanism == "MNAR") {
    return("step_indicate_na() + imputation # Preserve signal")
  }
}

Missing Data Mechanisms

Imputation Methods

# 1. Simple imputation (MCAR)
recipe(outcome ~ ., data = train) |>
  step_impute_median(all_numeric_predictors()) |>
  step_impute_mode(all_nominal_predictors())

# 2. Model-based imputation (MAR)
recipe(outcome ~ ., data = train) |>
  step_impute_knn(all_predictors(), neighbors = 5) |>
  # or
  step_impute_bag(all_predictors(), trees = 25)

# 3. Preserve missingness signal (MNAR)
recipe(outcome ~ ., data = train) |>
  step_indicate_na(all_predictors(), prefix = "missing_") |>
  step_impute_median(all_numeric_predictors())

See references/missing-data-strategies.md for:

• Diagnosing missing data mechanisms
• Advanced imputation techniques
• Multiple imputation
• Validation strategies

Complete Feature Engineering Workflow

Recommended Order

# Comprehensive feature engineering recipe
fe_recipe <- recipe(outcome ~ ., data = train) |>

  # 0. ROLE ASSIGNMENT
  update_role(id, new_role = "ID") |>

  # 1. MISSING DATA (if informative)
  step_indicate_na(all_predictors(), prefix = "missing_") |>
  step_impute_knn(all_numeric_predictors(), neighbors = 5) |>
  step_impute_mode(all_nominal_predictors()) |>

  # 2. FEATURE CREATION (before transformations)
  step_mutate(
    age_squared = age^2,
    income_per_person = income / household_size
  ) |>

  # 3. INTERACTION DETECTION (from previous analysis)
  step_interact(~ age:education + income:experience) |>

  # 4. NUMERIC TRANSFORMATIONS
  step_YeoJohnson(all_numeric_predictors()) |>

  # 5. CATEGORICAL ENCODING (choose method strategically)
  step_novel(all_nominal_predictors()) |>
  step_other(all_nominal_predictors(), threshold = 0.05) |>
  embed::step_lencode_mixed(high_cardinality_var, outcome = vars(outcome)) |>
  step_dummy(remaining_nominal_predictors()) |>

  # 6. STANDARDIZATION (after encoding)
  step_normalize(all_numeric_predictors()) |>

  # 7. FEATURE SELECTION
  step_zv(all_predictors()) |>
  step_corr(all_numeric_predictors(), threshold = 0.9) |>

  # 8. DIMENSIONALITY REDUCTION (if needed)
  step_pca(all_numeric_predictors(), threshold = 0.95)

Quick Reference: When to Use What

By Problem Type

High-dimensional (p > n)

• Filter methods → Lasso → validate

Small sample (n < 500)

• Be conservative with interactions
• Use penalized methods
• Validate rigorously

High-cardinality categoricals

• Target encoding or embeddings
• Feature hashing for very high

Imbalanced classes

• Consider interaction with class
• SMOTE before encoding

Time series / sequential

• Lag features
• Rolling statistics
• Seasonality indicators

By Model Type

Linear models

• Transform skewed predictors
• Standardize all numerics
• Consider splines for non-linearity

Tree-based

• Skip transformations
• Keep original scale
• Ordinal encoding OK

Neural networks

• Normalize to [0,1] or [-1,1]
• Consider embeddings for categoricals
• Interaction layers in architecture

SVM / KNN

• Standardization REQUIRED
• Remove correlations
• Consider PCA

Validation Best Practices

Preventing Data Leakage

# ❌ WRONG: Fit recipe on all data
rec_wrong <- recipe(outcome ~ ., data = all_data) |>
  step_normalize(all_numeric()) |>
  prep()

split <- initial_split(all_data)
train <- bake(rec_wrong, training(split))  # LEAKAGE!
test <- bake(rec_wrong, testing(split))

# ✅ CORRECT: Fit recipe only on training
split <- initial_split(all_data)

rec_correct <- recipe(outcome ~ ., data = training(split)) |>
  step_normalize(all_numeric()) |>
  prep(training = training(split))

train <- bake(rec_correct, new_data = NULL)  # Training data
test <- bake(rec_correct, new_data = testing(split))  # Test data

Cross-Validation for Feature Engineering

# All feature engineering inside CV
library(tidymodels)

# Recipe is NOT prepped outside CV
fe_recipe <- recipe(outcome ~ ., data = train) |>
  step_impute_knn(all_predictors()) |>
  step_normalize(all_numeric_predictors())

# Workflow bundles recipe + model
wf <- workflow() |>
  add_recipe(fe_recipe) |>  # Unprepped recipe
  add_model(linear_reg())

# CV fits recipe on each analysis set separately
cv_results <- fit_resamples(
  wf,
  resamples = vfold_cv(train, v = 10),
  metrics = metric_set(rmse, rsq)
)

# Collect honest performance estimates
collect_metrics(cv_results)

Supporting Files

Detailed References

• Categorical encoding: references/categorical-encoding.md
• Numeric transformations: references/numeric-transformations.md
• Interaction detection: references/interaction-detection.md
• Missing data strategies: references/missing-data-strategies.md
• Feature selection methods: references/feature-selection.md

Examples

• Encoding comparison: examples/encoding-comparison.md
• Interaction workflow: examples/interaction-workflow.md
• Selection pipeline: examples/selection-pipeline.md

Templates

• Feature engineering checklist: templates/feature-engineering-checklist.md

Resources

Primary Source

"Feature Engineering and Selection: A Practical Approach for Predictive Models"

• Authors: Max Kuhn & Kjell Johnson
• Website: https://feat.engineering/
• GitHub: https://github.com/topepo/FES
• License: Creative Commons (freely available)

Related Documentation

• tidymodels.org - Official tidymodels documentation
• recipes package - Feature engineering reference
• embed package - Advanced encoding methods

Relationship to Other Skills

Use this skill (r-feature-engineering) for:

• Strategic decisions about encoding methods
• Choosing transformation approaches
• Systematic interaction detection
• Feature selection strategy

Use r-tidymodels for:

• Implementing recipes and workflows
• Hyperparameter tuning
• Model comparison and selection
• Production deployment

Use r-datascience for:

• Exploratory data analysis
• Data wrangling and cleaning
• General ML workflow overview

Use tidyverse-expert for:

• dplyr data manipulation
• tidyr reshaping
• General data transformation