Very Simple Fela

Feature Engineering: Knowledge + Patterns

You are a feature engineering expert optimizing AUC on a tabular benchmark. Use the knowledge below however makes sense — there is no required workflow. Simple ideas may need one evaluation; complex ones may need systematic ablation.

Your goal: maximize AUC by iteratively building and evaluating feature sets via bench.evaluate(features_list). Stop when AUC stops improving (implement early-stop in your code).

First: detect which dataset you are on by checking bench.dataset (returns 'taobao' or 'dia'). Use the corresponding section below. Inspect bench.feature_columns_original and bench.X_train.dtypes to understand the actual schema — do not rely on skill hardcoded names.

Benchmark API

import sys
sys.path.insert(0, "benchmarks/feature_engineering")  # relative to AdaSkill project root
from feature_engineering import Benchmark, BenchmarkLimitReached

bench = Benchmark(dataset='<taobao or dia>', work_dir='<provided in prompt>', max_evaluations=500)
orig = bench.feature_columns_original.copy()  # inspect this first
dataset = bench.dataset   # 'taobao' or 'dia'

# Evaluate
auc = bench.evaluate(features_list)   # returns float AUC; raises BenchmarkLimitReached

Non-negotiable rules:

• Always modify both bench.X_train and bench.X_test when adding features
• For Taobao: compute CVR/count features from train_hist / valid_hist — never from X_train/X_test labels
• For Dia: all features derived directly from bench.X_train / bench.X_test columns (no history log)
• fillna() every new column before calling evaluate()
• Pass only columns that exist in bench.X_train to evaluate()

Early-stop (implement this in your code):

best_auc, no_improve = 0.0, 0
PATIENCE = 30   # from --early-stop arg passed in prompt

while True:
    auc = bench.evaluate(features)
    if auc > best_auc:
        best_auc = auc
        no_improve = 0
    else:
        no_improve += 1
    if no_improve >= PATIENCE:
        break

Taobao Dataset

Ad conversion prediction. Target column indicates whether a purchase occurred (~1.9% positive rate).

The dataset contains features in these semantic groups — inspect the actual column names via bench.feature_columns_original:

• User entity: user ID, demographic attributes (age level, gender, occupation, star level)
• Item entity: item ID, category, brand, city, price level, sales level, collection level, page-view level, property count
• Shop entity: shop ID, review count level, positive review rate, star level, service/delivery/description scores
• Context: timestamp, page ID (extracted hour and day are also available)
• Prediction signal: predicted category/property match features

Time split: history log covers a train window and a validation window (sequential days).

train_hist = bench.train_history_df   # history log — safe to use (no leakage)
valid_hist  = bench.valid_history_df
TARGET = bench.target_column          # purchase indicator column name
G_CVR   = train_hist[TARGET].mean()
G_CVR_V = valid_hist[TARGET].mean()

Taobao: Feature Engineering Directions

This is a sparse, high-cardinality CTR/CVR prediction task. The most valuable signal comes from behavioral statistics computed from the history log.

Productive directions:

• Entity CVR: For each entity type (user, item, shop, category, brand), compute its historical conversion rate from the history log. Bayesian smoothing prevents overfitting on rare entities.
• Interaction CVR: CVR conditioned on two entities jointly (e.g., user × category, user × shop) captures preference alignment beyond marginal rates.
• Frequency/activity counts: How often does each entity appear in history? Log-transform these — count distributions are heavy-tailed.
• Price deviation: Whether this item is cheap or expensive relative to what this user typically sees — captures price sensitivity.
• Quality composites: The item has multiple popularity-related ordinal features (sales, collection, page-views); combining them into a single composite reduces noise. Similarly for shop quality scores.
• Temporal features: Hour of day and day extracted from the timestamp — useful when combined with CVR, weak alone.

Taobao: Code Patterns

# Bayesian-smoothed CVR — core pattern for any high-cardinality entity column
def add_cvr(key_col, feat_name, k=20):
    """k controls smoothing: larger k = more shrinkage toward global mean.
    Use larger k for 1-way, smaller k for 2-way (fewer samples per cell)."""
    def _cvr(hist, g):
        s = hist.groupby(key_col)[TARGET].agg(['sum', 'count'])
        s['v'] = (s['sum'] + k * g) / (s['count'] + k)
        return s['v'].to_dict()
    tr = _cvr(train_hist, G_CVR)
    va = _cvr(valid_hist, G_CVR_V)
    bench.X_train[feat_name] = bench.X_train[key_col].map(tr).fillna(G_CVR)
    bench.X_test[feat_name]  = bench.X_test[key_col].map(va).fillna(G_CVR_V)

# Log-count frequency — how active is this entity in history?
def add_count(key_col, feat_name):
    tr = train_hist[key_col].value_counts().apply(np.log1p).to_dict()
    va = valid_hist[key_col].value_counts().apply(np.log1p).to_dict()
    bench.X_train[feat_name] = bench.X_train[key_col].map(tr).fillna(0)
    bench.X_test[feat_name]  = bench.X_test[key_col].map(va).fillna(0)

# Numeric aggregation from history (e.g., user's average price preference)
def add_agg(group_col, value_col, agg_fn, feat_name):
    tr = train_hist.groupby(group_col)[value_col].agg(agg_fn).to_dict()
    va = valid_hist.groupby(group_col)[value_col].agg(agg_fn).to_dict()
    fb_tr = train_hist[value_col].agg(agg_fn)
    fb_va = valid_hist[value_col].agg(agg_fn)
    bench.X_train[feat_name] = bench.X_train[group_col].map(tr).fillna(fb_tr)
    bench.X_test[feat_name]  = bench.X_test[group_col].map(va).fillna(fb_va)

# Cross-feature product of two CVR signals
# bench.X_train['user_x_cat_cvr'] = bench.X_train['user_cvr'] * bench.X_train['cat_cvr']
# bench.X_test['user_x_cat_cvr']  = bench.X_test['user_cvr']  * bench.X_test['cat_cvr']

Taobao: Engineering Principles

1. Raw high-cardinality ID columns become redundant once you have their CVR. The entity ID columns (user, item, shop, brand, city) as raw integers carry almost no signal for tree models once their corresponding CVR features are computed. Explicitly exclude them from FEATURES — keeping them adds noise. This is a standard principle in CTR feature engineering, not dataset-specific.
2. Ratio/division needs protection. Always use denominator + 1e-6. Groupby means on sparse keys: fillna with global mean.
3. Log-transform count features. Raw counts are heavy-tailed and hurt model calibration.
4. Smoothing strength scales with data density. Denser keys tolerate less smoothing; sparse 2-way combination keys need more smoothing.
5. Composite scores reduce noise. Summing a group of related ordinal features is usually better than using each separately.
6. Ablate from a strong set. Once you have a promising feature set, try removing features one at a time — lean sets often generalize better.

Dia Dataset

Diabetes risk prediction based on a health survey. Target column is a binary diabetes indicator. All original features are binary (0/1) or low-cardinality ordinal integers. No history log — all features derived directly from X_train/X_test columns.

Inspect the actual column names via bench.feature_columns_original. The dataset contains features in these semantic groups:

• Metabolic/cardiovascular risk indicators: binary flags for high blood pressure, high cholesterol, cholesterol check, heart disease/attack history, stroke history
• BMI: continuous body mass index
• Lifestyle behaviors: binary flags for smoking, physical activity, fruit/vegetable consumption, heavy alcohol consumption
• Healthcare access: binary flags for healthcare coverage, cost-related doctor avoidance
• Functional health: ordinal general health status (1–5), days of poor physical/mental health (0–30), walking difficulty
• Sociodemographic: sex (binary), age category (ordinal), education level (ordinal), income level (ordinal)

Dia: Feature Engineering Directions

The original features are already clinically meaningful, so gains come from capturing non-linearity and interactions between them.

Productive directions:

• BMI non-linearity: BMI has well-known clinical thresholds (overweight ≥25, obese ≥30) with a non-linear relationship to metabolic risk. Threshold indicators and polynomial terms capture this.
• Cardiovascular composite: Blood pressure, heart disease, stroke, and cholesterol indicators share common pathological mechanisms (insulin resistance, inflammation). Summing them into a composite score can be more informative than using each separately.
• Lifestyle composite: Smoking, physical inactivity, heavy alcohol, and poor diet interact synergistically — a cumulative unhealthy lifestyle score captures joint behavioral risk.
• Health burden composite: General health status, days of poor physical/mental health, and walking difficulty all reflect functional health — combining them is a natural direction.
• Target encoding of ordinals: Age, general health, income, and education are ordinal but their relationship with diabetes risk may not be strictly linear. Cross-validated target encoding captures the true pattern without leakage.
• Pairwise interaction encodings: Combinations of age × general health, age × blood pressure, etc. may capture interaction effects worth encoding via cross-validated target encoding.
• Socioeconomic interactions: Income, education, and healthcare access interact — lower socioeconomic status affects both lifestyle choices and medical access.
• Age × sex interaction: Diabetes risk has different age trajectories by sex (e.g., post-menopausal elevated risk). An interaction term can capture this.

Important: The baseline is already high. Each feature must earn its place. A clean set of 20–30 well-chosen features typically beats 100+ noisy ones.

Dia: Code Patterns

# Cross-validated target encoding — prevents leakage from target variable
from sklearn.model_selection import KFold

def cv_target_encode(X_tr, X_te, col, y_tr, n_splits=5):
    """Encode ordinal/categorical column by per-fold mean target rate.
    KFold prevents target leakage on the training set."""
    global_mean = y_tr.mean()
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)
    train_enc = np.zeros(len(X_tr))
    for tr_idx, val_idx in kf.split(X_tr):
        fold_mean = (X_tr.iloc[tr_idx]
                     .assign(_y=y_tr.iloc[tr_idx])
                     .groupby(col)['_y'].mean())
        train_enc[val_idx] = X_tr.iloc[val_idx][col].map(fold_mean).fillna(global_mean)
    full_mean = X_tr.assign(_y=y_tr).groupby(col)['_y'].mean()
    test_enc = X_te[col].map(full_mean).fillna(global_mean)
    return train_enc, test_enc

y_train = bench.y_train  # target labels — safe to use for computing train-side encodings

# Encode ordinal columns by their mean target rate (identify them from bench.feature_columns_original)
# for col in <ordinal columns>:
#     tr_enc, te_enc = cv_target_encode(bench.X_train, bench.X_test, col, y_train)
#     bench.X_train[f'{col}_te'] = tr_enc
#     bench.X_test[f'{col}_te']  = te_enc

# BMI threshold indicators (WHO/CDC clinical standards — not data-derived)
# for df in [bench.X_train, bench.X_test]:
#     df['bmi_overweight'] = (df[bmi_col] >= 25).astype(int)
#     df['bmi_obese']      = (df[bmi_col] >= 30).astype(int)
#     df['bmi_sq']         = df[bmi_col] ** 2

# Composite scores — sum binary indicators that share a clinical mechanism
# for df in [bench.X_train, bench.X_test]:
#     df['cardio_composite']    = df[bp_col] + df[heart_col] + df[stroke_col] + df[chol_col]
#     df['lifestyle_risk']      = df[smoker_col] + (1 - df[activity_col]) + df[alcohol_col] + ...
#     df['health_burden']       = df[physhlth_col] + df[menthlth_col] + df[diffwalk_col] * 5

# Polynomial terms for non-linear ordinals
# for df in [bench.X_train, bench.X_test]:
#     df['age_sq']      = df[age_col] ** 2
#     df['genhlth_sq']  = df[genhlth_col] ** 2
#     df['age_x_genhlth'] = df[age_col] * df[genhlth_col]

# Interaction target encoding: build combo key, encode, drop string column
# for col_a, col_b in [<pair1>, <pair2>]:
#     combo = f'{col_a}_x_{col_b}'
#     bench.X_train[combo] = bench.X_train[col_a].astype(str) + '_' + bench.X_train[col_b].astype(str)
#     bench.X_test[combo]  = bench.X_test[col_a].astype(str)  + '_' + bench.X_test[col_b].astype(str)
#     tr_enc, te_enc = cv_target_encode(bench.X_train, bench.X_test, combo, y_train)
#     bench.X_train[f'{combo}_te'] = tr_enc
#     bench.X_test[f'{combo}_te']  = te_enc
#     bench.X_train.drop(columns=[combo], inplace=True)
#     bench.X_test.drop(columns=[combo], inplace=True)

Dia: Engineering Principles

1. Always use KFold target encoding, never fit-on-full-train. Encoding ordinals without cross-validation leaks the target into training features and inflates AUC artificially.
2. BMI thresholds are clinically grounded, not data-mined. The ≥25 (overweight) and ≥30 (obese) cutoffs come from WHO/CDC standards.
3. Composite scores capture synergy. Multiple co-occurring risk factors (cardiovascular, lifestyle) have cumulative effects — a sum composite is often more informative than each factor individually.
4. Log-transform day-count features. The 0–30 count features for days of poor health are right-skewed; log1p stabilizes variance.
5. Non-linearity in ordinals matters. Age and general health status have non-linear effects; squared terms and target encoding both worth trying.
6. Lean sets outperform large ones here. The baseline is already high; noise features actively hurt. Test removal of features that don't contribute.
7. Ratio features need protection. Any division: use denominator + 1e-6.