| name | predictive_modeling |
| description | Build, evaluate, calibrate, and operationalize supervised prediction models inside the Dataclaw data-science workflow with leakage-safe validation, decision-aligned metrics and thresholds, honest baselines, subgroup and shift checks, uncertainty, and deployment monitoring. Use for classification, regression, ranking, propensity or risk scores, churn and fraud prediction, lead scoring, probability calibration, model comparison, model cards, and audits of existing predictive systems. |
| tags | ["modeling","machine-learning","prediction","validation","method"] |
Related skills: dataclaw_data_science (governed workflow), structured_eda (target and population readiness), feature_engineering (model inputs), forecasting (time-indexed prediction), causal_inference (effect claims), segmentation (typing and value/risk tiers), analysis_review (validation gate), visualization, report_design, and artifacts (delivery).
Predictive Modeling Playbook
Run prediction inside dataclaw_data_science; this skill owns target framing, split design, baselines, evaluation, selection, calibration, and operational monitoring. It does not authorize causal claims. Standard model training runs in the notebook with libraries such as scikit-learn, xgboost, or lightgbm; do not invent a predictive-modeling tool.
Follow this process:
- Define the decision, prediction unit, target event/value, observation window, prediction time, forecast horizon, eligible population, action capacity, error costs, and how a score changes an action. If no action or evaluation population can be named, do not optimize a model yet.
- Discover data, open a notebook, and fetch
structured_eda. Audit label provenance, censoring/delay, prevalence, repeated entities, temporal drift, missing-label selection, leakage candidates, identifiers, protected attributes, and mismatch between training and deployment populations.
- Choose the task and evaluation route from the table. Use
forecasting when the primary dependence is time-series forecasting and causal_inference when the question is intervention impact.
- Propose the plan and wait for approval. State the target contract, feature availability cutoff, split scheme, frozen test set, baseline, candidate families, tuning budget, primary decision metric, minimum improvement, threshold/capacity rule, calibration, subgroup/fairness checks, uncertainty, and refusal conditions.
- Fetch
feature_engineering. Put every learned transform, selection step, resampling step, and target-derived encoding inside the training fold. Exclude direct identifiers; treat protected attributes as audit variables unless their use is explicitly justified and governed.
- Fit a naive/business baseline and a transparent linear/tree baseline before complex candidates. Evaluate all candidates on identical folds and log comparable MLflow runs with seeds, data/version identifiers, parameters, metrics, and fitted pipelines.
- Select using cross-validation or nested cross-validation without touching the final test. Freeze the model and threshold, then evaluate once on a deployment-mirroring holdout; do not repeatedly tune against it.
- Report discrimination/error, calibration when scores are probabilistic, performance at the operating threshold or capacity, uncertainty, subgroup results, coverage/abstention, and improvement over baseline. AUC alone never establishes operational value.
- Stress deployment: temporal/out-of-domain slices, plausible feature drift, missing inputs, extreme values, and delayed labels. Define monitoring for input drift, score drift, calibration/performance decay, outcome availability, threshold volume, and retraining triggers.
- Request analysis review and deliver an evidence-bound model card/report. Do not ship the model when the target is invalid, the test set is contaminated, key groups have unmeasured performance, or deployment inputs cannot reproduce training features.
Task and metric routing
| Decision | Primary evaluation | Required caveat |
|---|
| Rank a limited queue | Precision/recall or gain/lift at capacity; PR-AUC | Evaluate at the actual review/contact capacity |
| Produce probabilities | Log loss/Brier plus calibration curve/slope | Recalibrate only on validation data; audit by subgroup |
| Binary action with asymmetric costs | Expected utility/cost at a frozen threshold | State false-positive/false-negative costs and prevalence |
| Continuous prediction | MAE/RMSE/quantile loss in decision units | Inspect heteroskedasticity and tail errors |
| Rare-event detection | PR-AUC, recall at alert budget, false positives per unit time | Accuracy and ROC-AUC can hide operational burden |
| Ordinal outcome | Ordinal loss, weighted kappa, threshold metrics | Do not flatten order into unrelated classes |
| Time-indexed future values | forecasting | Use rolling-origin evaluation and time-aware baselines |
Validation and selection contract
- Split by the leakage unit: group repeated users/accounts, split forward for temporal deployment, and preserve site/geography boundaries when transport is the risk.
- Use nested CV when tuning/search is material and data is limited. Report fold spread or bootstrap intervals; a single score is not evidence.
- Compare against prevalence/mean, current business rule, and a simple interpretable model. Complexity must produce decision-relevant improvement, not merely a tiny metric gain.
- Select the operating threshold from validation data using declared costs, capacity, or sensitivity constraints. Lock it before final test evaluation and report confusion-matrix counts/rates at that point.
- Calibrate on held-out or cross-fitted predictions. Report reliability, not only discrimination; recalibration does not repair covariate or concept shift.
- For imbalanced data, resample or weight only inside training folds and evaluate on the natural deployment prevalence.
- Use subgroup metrics with adequate uncertainty and intersectional checks when decisions affect people. Do not claim fairness from aggregate parity or suppress evidence of poor performance in a small group.
- Prefer interpretable diagnostics appropriate to the model; feature importance and SHAP are predictive associations, not causal explanations.
Threats to validity
- Label leakage and target proxies — audit availability at prediction time and remove post-outcome fields.
- Label selection/censoring — model the eligible labeled population and bound claims when outcomes are missing selectively.
- Entity/time contamination — group- or time-aware splits prevent memorization and look-ahead.
- Validation overfitting — nested tuning plus a single frozen final test.
- Prevalence/threshold mismatch — evaluate at deployment prevalence, costs, and capacity.
- Distribution and concept shift — stress slices and define monitoring/retraining triggers.
- Feedback loops — predictions change who receives an action and therefore which labels return; retain exploration/audit samples.
- Proxy discrimination — keep protected attributes for auditing, exclude or govern their use, and inspect correlated proxies.
Domain controls
An opt-in profile is a technical control bundle, not proof of compliance. For credit, pricing, employment, healthcare, safety, or fraud, require qualified human review, contestability/override paths, subgroup uncertainty, and explicit abstention/escalation. Keep identifiers and raw sensitive rows out of model-visible output; suppress every sensitive output cell below a default 5 units (raise but never lower the floor) with complementary suppression so visible totals cannot reconstruct it. Never auto-deploy or change a consequential threshold without explicit approval.
