菜单
Use when preparing raw data for modeling — handling missing values, duplicates, inconsistent types, outliers, and bad categorical values. Emphasizes fitting all imputation on train-only to avoid leakage.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | data-cleaning |
| description | Use when preparing raw data for modeling — handling missing values, duplicates, inconsistent types, outliers, and bad categorical values. Emphasizes fitting all imputation on train-only to avoid leakage. |
Cleaning turns raw data into a consistent, model-ready table without leaking information from the future or the test set. The golden rule: every statistic used to clean (means, medians, modes, bounds, category maps) must be learned from the training split only, then applied to validation/test.
| Situation | Strategy |
|---|---|
| Numeric, MCAR, small % | Median impute (robust to skew) |
| Numeric, informative missingness | Impute + add was_missing indicator |
| Categorical | Impute with "Missing" as its own category |
| Time series | Forward/backward fill within group |
| >50% missing | Consider dropping the column |
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
# Fit imputers on TRAIN ONLY
num_imputer = SimpleImputer(strategy="median").fit(X_train[num_cols])
X_train[num_cols] = num_imputer.transform(X_train[num_cols])
X_test[num_cols] = num_imputer.transform(X_test[num_cols]) # reuse train stats
Prefer doing this inside a Pipeline/ColumnTransformer (see the sklearn-pipelines skill) so leakage is impossible by construction.
# Winsorize numeric columns to train-derived 1st/99th percentiles
lo = X_train[col].quantile(0.01)
hi = X_train[col].quantile(0.99)
X_train[col] = X_train[col].clip(lo, hi)
X_test[col] = X_test[col].clip(lo, hi)
df.fillna(df.mean()) on the whole dataset — classic leakage. Compute stats on train only.dropna() on rows silently shrinks and biases the dataset; prefer imputation + indicators.Produce a clean, typed dataframe plus a documented list of decisions (what was imputed/capped/dropped and why) so feature-engineering and reproducible-ml can rely on it.
Use when running ML experiments that need to be compared, reproduced, or shared. Covers MLflow/Weights & Biases logging, what to track, run organization, and model registry basics.
Use when starting on a new dataset, before modeling, or when asked to "explore", "profile", "understand", or "summarize" data. Covers structured EDA, distributions, correlations, target leakage checks, and visualization.
Use when creating, encoding, scaling, or selecting features for ML models. Covers categorical encoding, numeric transforms, datetime/text/aggregation features, and leakage-safe target encoding.
Use when optimizing model hyperparameters. Covers search strategy (random vs Bayesian/Optuna), leakage-safe tuning inside CV, search-space design, early stopping, and budget management.
Use when the target is rare (fraud, churn, disease, anomalies). Covers correct metrics, resampling (SMOTE/undersampling), class weights, threshold tuning, and avoiding the accuracy trap and resampling leakage.
Use when fine-tuning a large language model. Covers choosing full vs LoRA/QLoRA, dataset formatting, the transformers/PEFT/TRL stack, key hyperparameters, and evaluating fine-tunes without overfitting.