메뉴
Use when choosing metrics, validating models, or interpreting results. Covers metric selection by problem type, cross-validation strategy, calibration, confusion-matrix analysis, and avoiding misleading scores.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | model-evaluation |
| description | Use when choosing metrics, validating models, or interpreting results. Covers metric selection by problem type, cross-validation strategy, calibration, confusion-matrix analysis, and avoiding misleading scores. |
The wrong metric on the wrong split produces confident, wrong conclusions. Evaluation is about choosing a metric that matches the business cost, validating it on a split that mirrors production, and reporting it honestly with uncertainty.
| Problem | Default metric | Use when |
|---|---|---|
| Balanced classification | ROC-AUC, accuracy | classes ~balanced |
| Imbalanced classification | PR-AUC, F1, recall@k | rare positives (fraud, disease) |
| Probabilistic output | Log loss, Brier, calibration | you need trustworthy probabilities |
| Ranking | NDCG, MAP, MRR | recommendation/search |
| Regression | MAE (robust), RMSE (penalize big errors) | match error cost |
| Regression, multiplicative | MAPE / RMSLE | errors scale with magnitude |
StratifiedKFold for classification.TimeSeriesSplit — never shuffle; train on past, validate on future.GroupKFold so the same group never spans train and test.from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np
cv = StratifiedKFold(5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring="average_precision")
print(f"PR-AUC: {scores.mean():.3f} ± {scores.std():.3f}") # always report spread
CalibratedClassifierCV or reliability curves when probabilities feed decisions.A defensible metric + validated score with uncertainty, plus a chosen decision threshold, for experiment-tracking to log and stakeholders to trust.
Use when preparing raw data for modeling — handling missing values, duplicates, inconsistent types, outliers, and bad categorical values. Emphasizes fitting all imputation on train-only to avoid leakage.
Use when running ML experiments that need to be compared, reproduced, or shared. Covers MLflow/Weights & Biases logging, what to track, run organization, and model registry basics.
Use when starting on a new dataset, before modeling, or when asked to "explore", "profile", "understand", or "summarize" data. Covers structured EDA, distributions, correlations, target leakage checks, and visualization.
Use when creating, encoding, scaling, or selecting features for ML models. Covers categorical encoding, numeric transforms, datetime/text/aggregation features, and leakage-safe target encoding.
Use when optimizing model hyperparameters. Covers search strategy (random vs Bayesian/Optuna), leakage-safe tuning inside CV, search-space design, early stopping, and budget management.
Use when the target is rare (fraud, churn, disease, anomalies). Covers correct metrics, resampling (SMOTE/undersampling), class weights, threshold tuning, and avoiding the accuracy trap and resampling leakage.