very-simple-fela

# Feature Engineering: Knowledge + Patterns You are a feature engineering expert optimizing AUC on a tabular benchmark. Use the knowledge below however makes sense — there is no required workflow. Simple ideas may need one evaluation; complex ones may need systematic ablation. Your goal: maximize AUC by iteratively building and evaluating feature sets via `bench.evaluate(features_list)`. Stop when AUC stops improving (implement early-stop in your code). **First: detect which dataset you are on** by checking `bench.dataset` (returns `'taobao'` or `'dia'`). Use the corresponding section below. Inspect `bench.feature_columns_original` and `bench.X_train.dtypes` to understand the actual schema — do not rely on skill hardcoded names. --- ## Benchmark API ```python import sys sys.path.insert(0, "benchmarks/feature_engineering") # relative to AdaSkill project root from feature_engineering import Benchmark, BenchmarkLimitReached bench = Benchmark(dataset='<taobao or dia>', work_dir='<provided in prompt>', max_evaluations=500) orig = bench.feature_columns_original.copy() # inspect this first dataset = bench.dataset # 'taobao' or 'dia' # Evaluate auc = bench.evaluate(features_list) # returns float AUC; raises BenchmarkLimitReached ``` **Non-negotiable rules:** - Always modify **both** `bench.X_train` and `bench.X_test` when adding features - For Taobao: compute CVR/count features from `train_hist` / `valid_hist` — never from X_train/X_test labels - For Dia: all features derived directly from `bench.X_train` / `bench.X_test` columns (no history log) - `fillna()` every new column before calling `evaluate()` - Pass only columns that exist in `bench.X_train` to `evaluate()` **Early-stop (implement this in your code):** ```python best_auc, no_improve = 0.0, 0 PATIENCE = 30 # from --early-stop arg passed in prompt while True: auc = bench.evaluate(features) if auc > best_auc: best_auc = auc no_improve = 0 else: no_improve += 1 if no_improve >= PATIENCE: break ``` --- ## Taobao Dataset Ad conversion prediction. Target column indicates whether a purchase occurred (~1.9% positive rate). The dataset contains features in these semantic groups — inspect the actual column names via `bench.feature_columns_original`: - **User entity**: user ID, demographic attributes (age level, gender, occupation, star level) - **Item entity**: item ID, category, brand, city, price level, sales level, collection level, page-view level, property count - **Shop entity**: shop ID, review count level, positive review rate, star level, service/delivery/description scores - **Context**: timestamp, page ID (extracted hour and day are also available) - **Prediction signal**: predicted category/property match features **Time split:** history log covers a train window and a validation window (sequential days). ```python train_hist = bench.train_history_df # history log — safe to use (no leakage) valid_hist = bench.valid_history_df TARGET = bench.target_column # purchase indicator column name G_CVR = train_hist[TARGET].mean() G_CVR_V = valid_hist[TARGET].mean() ``` ### Taobao: Feature Engineering Directions This is a sparse, high-cardinality CTR/CVR prediction task. The most valuable signal comes from **behavioral statistics** computed from the history log. **Productive directions:** - **Entity CVR**: For each entity type (user, item, shop, category, brand), compute its historical conversion rate from the history log. Bayesian smoothing prevents overfitting on rare entities. - **Interaction CVR**: CVR conditioned on two entities jointly (e.g., user × category, user × shop) captures preference alignment beyond marginal rates. - **Frequency/activity counts**: How often does each entity appear in history? Log-transform these — count distributions are heavy-tailed. - **Price deviation**: Whether this item is cheap or expensive relative to what this user typically sees — captures price sensitivity. - **Quality composites**: The item has multiple popularity-related ordinal features (sales, collection, page-views); combining them into a single composite reduces noise. Similarly for shop quality scores. - **Temporal features**: Hour of day and day extracted from the timestamp — useful when combined with CVR, weak alone. ### Taobao: Code Patterns ```python # Bayesian-smoothed CVR — core pattern for any high-cardinality entity column def add_cvr(key_col, feat_name, k=20): """k controls smoothing: larger k = more shrinkage toward global mean. Use larger k for 1-way, smaller k for 2-way (fewer samples per cell).""" def _cvr(hist, g): s = hist.groupby(key_col)[TARGET].agg(['sum', 'count']) s['v'] = (s['sum'] + k * g) / (s['count'] + k) return s['v'].to_dict() tr = _cvr(train_hist, G_CVR) va = _cvr(valid_hist, G_CVR_V) bench.X_train[feat_name] = bench.X_train[key_col].map(tr).fillna(G_CVR) bench.X_test[feat_name] = bench.X_test[key_col].map(va).fillna(G_CVR_V) # Log-count frequency — how active is this entity in history? def add_count(key_col, feat_name): tr = train_hist[key_col].value_counts().apply(np.log1p).to_dict() va = valid_hist[key_col].value_counts().apply(np.log1p).to_dict() bench.X_train[feat_name] = bench.X_train[key_col].map(tr).fillna(0) bench.X_test[feat_name] = bench.X_test[key_col].map(va).fillna(0) # Numeric aggregation from history (e.g., user's average price preference) def add_agg(group_col, value_col, agg_fn, feat_name): tr = train_hist.groupby(group_col)[value_col].agg(agg_fn).to_dict() va = valid_hist.groupby(group_col)[value_col].agg(agg_fn).to_dict() fb_tr = train_hist[value_col].agg(agg_fn) fb_va = valid_hist[value_col].agg(agg_fn) bench.X_train[feat_name] = bench.X_train[group_col].map(tr).fillna(fb_tr) bench.X_test[feat_name] = bench.X_test[group_col].map(va).fillna(fb_va) # Cross-feature product of two CVR signals # bench.X_train['user_x_cat_cvr'] = bench.X_train['user_cvr'] * bench.X_train['cat_cvr'] # bench.X_test['user_x_cat_cvr'] = bench.X_test['user_cvr'] * bench.X_test['cat_cvr'] ``` ### Taobao: Engineering Principles 1. **Raw high-cardinality ID columns become redundant once you have their CVR.** The entity ID columns (user, item, shop, brand, city) as raw integers carry almost no signal for tree models once their corresponding CVR features are computed. Explicitly exclude them from `FEATURES` — keeping them adds noise. This is a standard principle in CTR feature engineering, not dataset-specific. 2. **Ratio/division needs protection.** Always use `denominator + 1e-6`. Groupby means on sparse keys: fillna with global mean. 3. **Log-transform count features.** Raw counts are heavy-tailed and hurt model calibration. 4. **Smoothing strength scales with data density.** Denser keys tolerate less smoothing; sparse 2-way combination keys need more smoothing. 5. **Composite scores reduce noise.** Summing a group of related ordinal features is usually better than using each separately. 6. **Ablate from a strong set.** Once you have a promising feature set, try removing features one at a time — lean sets often generalize better. --- ## Dia Dataset Diabetes risk prediction based on a health survey. Target column is a binary diabetes indicator. All original features are binary (0/1) or low-cardinality ordinal integers. **No history log — all features derived directly from X_train/X_test columns.** Inspect the actual column names via `bench.feature_columns_original`. The dataset contains features in these semantic groups: - **Metabolic/cardiovascular risk indicators**: binary flags for high blood pressure, high cholesterol, cholesterol check, heart disease/attack history, stroke history - **BMI**: continuous body mass index - **Lifestyle behaviors**: binary flags for smoking, physical activity, fruit/vegetable consumption, heavy alcohol consumption - **Healthcare access**: binary flags for healthcare coverage, cost-related doctor avoidance - **Functional health**: ordinal general health status (1–5), days of poor physical/mental health (0–30), walking difficulty - **Sociodemographic**: sex (binary), age category (ordinal), education level (ordinal), income level (ordinal) ### Dia: Feature Engineering Directions The original features are already clinically meaningful, so gains come from **capturing non-linearity and interactions** between them. **Productive directions:** - **BMI non-linearity**: BMI has well-known clinical thresholds (overweight ≥25, obese ≥30) with a non-linear relationship to metabolic risk. Threshold indicators and polynomial terms capture this. - **Cardiovascular composite**: Blood pressure, heart disease, stroke, and cholesterol indicators share common pathological mechanisms (insulin resistance, inflammation). Summing them into a composite score can be more informative than using each separately. - **Lifestyle composite**: Smoking, physical inactivity, heavy alcohol, and poor diet interact synergistically — a cumulative unhealthy lifestyle score captures joint behavioral risk. - **Health burden composite**: General health status, days of poor physical/mental health, and walking difficulty all reflect functional health — combining them is a natural direction. - **Target encoding of ordinals**: Age, general health, income, and education are ordinal but their relationship with diabetes risk may not be strictly linear. Cross-validated target encoding captures the true pattern without leakage. - **Pairwise interaction encodings**: Combinations of age × general health, age × blood pressure, etc. may capture interaction effects worth encoding via cross-validated target encoding. - **Socioeconomic interactions**: Income, education, and healthcare access interact — lower socioeconomic status affects both lifestyle choices and medical access. - **Age × sex interaction**: Diabetes risk has different age trajectories by sex (e.g., post-menopausal elevated risk). An interaction term can capture this. **Important:** The baseline is already high. Each feature must earn its place. A clean set of 20–30 well-chosen features typically beats 100+ noisy ones. ### Dia: Code Patterns ```python # Cross-validated target encoding — prevents leakage from target variable from sklearn.model_selection import KFold def cv_target_encode(X_tr, X_te, col, y_tr, n_splits=5): """Encode ordinal/categorical column by per-fold mean target rate. KFold prevents target leakage on the training set.""" global_mean = y_tr.mean() kf = KFold(n_splits=n_splits, shuffle=True, random_state=42) train_enc = np.zeros(len(X_tr)) for tr_idx, val_idx in kf.split(X_tr): fold_mean = (X_tr.iloc[tr_idx] .assign(_y=y_tr.iloc[tr_idx]) .groupby(col)['_y'].mean()) train_enc[val_idx] = X_tr.iloc[val_idx][col].map(fold_mean).fillna(global_mean) full_mean = X_tr.assign(_y=y_tr).groupby(col)['_y'].mean() test_enc = X_te[col].map(full_mean).fillna(global_mean) return train_enc, test_enc y_train = bench.y_train # target labels — safe to use for computing train-side encodings # Encode ordinal columns by their mean target rate (identify them from bench.feature_columns_original) # for col in <ordinal columns>: # tr_enc, te_enc = cv_target_encode(bench.X_train, bench.X_test, col, y_train) # bench.X_train[f'{col}_te'] = tr_enc # bench.X_test[f'{col}_te'] = te_enc # BMI threshold indicators (WHO/CDC clinical standards — not data-derived) # for df in [bench.X_train, bench.X_test]: # df['bmi_overweight'] = (df[bmi_col] >= 25).astype(int) # df['bmi_obese'] = (df[bmi_col] >= 30).astype(int) # df['bmi_sq'] = df[bmi_col] ** 2 # Composite scores — sum binary indicators that share a clinical mechanism # for df in [bench.X_train, bench.X_test]: # df['cardio_composite'] = df[bp_col] + df[heart_col] + df[stroke_col] + df[chol_col] # df['lifestyle_risk'] = df[smoker_col] + (1 - df[activity_col]) + df[alcohol_col] + ... # df['health_burden'] = df[physhlth_col] + df[menthlth_col] + df[diffwalk_col] * 5 # Polynomial terms for non-linear ordinals # for df in [bench.X_train, bench.X_test]: # df['age_sq'] = df[age_col] ** 2 # df['genhlth_sq'] = df[genhlth_col] ** 2 # df['age_x_genhlth'] = df[age_col] * df[genhlth_col] # Interaction target encoding: build combo key, encode, drop string column # for col_a, col_b in [<pair1>, <pair2>]: # combo = f'{col_a}_x_{col_b}' # bench.X_train[combo] = bench.X_train[col_a].astype(str) + '_' + bench.X_train[col_b].astype(str) # bench.X_test[combo] = bench.X_test[col_a].astype(str) + '_' + bench.X_test[col_b].astype(str) # tr_enc, te_enc = cv_target_encode(bench.X_train, bench.X_test, combo, y_train) # bench.X_train[f'{combo}_te'] = tr_enc # bench.X_test[f'{combo}_te'] = te_enc # bench.X_train.drop(columns=[combo], inplace=True) # bench.X_test.drop(columns=[combo], inplace=True) ``` ### Dia: Engineering Principles 1. **Always use KFold target encoding, never fit-on-full-train.** Encoding ordinals without cross-validation leaks the target into training features and inflates AUC artificially. 2. **BMI thresholds are clinically grounded, not data-mined.** The ≥25 (overweight) and ≥30 (obese) cutoffs come from WHO/CDC standards. 3. **Composite scores capture synergy.** Multiple co-occurring risk factors (cardiovascular, lifestyle) have cumulative effects — a sum composite is often more informative than each factor individually. 4. **Log-transform day-count features.** The 0–30 count features for days of poor health are right-skewed; log1p stabilizes variance. 5. **Non-linearity in ordinals matters.** Age and general health status have non-linear effects; squared terms and target encoding both worth trying. 6. **Lean sets outperform large ones here.** The baseline is already high; noise features actively hurt. Test removal of features that don't contribute. 7. **Ratio features need protection.** Any division: use `denominator + 1e-6`.