python-ml-predictive

name	python-ml-predictive
description	Use when adding forecasting, classification, regression, or anomaly detection to a SaaS feature — demand/sales/cash-flow forecasting, churn and risk scoring, anomaly detection — with scikit-learn, Prophet, and statsmodels. Covers data prep, model serving, monitoring, and explainability.
metadata	{"portable":true,"compatible_with":["Codex","codex"]}

Python ML & Predictive Analytics

Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, [phone].

Use When

• Use when adding forecasting, classification, regression, or anomaly detection to a SaaS feature — demand/sales/cash-flow forecasting, churn and risk scoring, anomaly detection — with scikit-learn, Prophet, and statsmodels. Covers data prep, model serving, monitoring, and explainability.
• The task needs reusable judgment, domain constraints, or a proven workflow rather than ad hoc advice.

Do Not Use When

• The task is unrelated to python-ml-predictive or would be better handled by a more specific companion skill.
• The request only needs a trivial answer and none of this skill's constraints or references materially help.

Required Inputs

• Gather relevant project context, constraints, and the concrete problem to solve; load references only as needed.
• Confirm the desired deliverable: design, code, review, migration plan, audit, or documentation.

Workflow

• Read this SKILL.md first, then load only the referenced deep-dive files that are necessary for the task.
• Apply the ordered guidance, checklists, and decision rules in this skill instead of cherry-picking isolated snippets.
• Produce the deliverable with assumptions, risks, and follow-up work made explicit when they matter.

Quality Standards

• Keep outputs execution-oriented, concise, and aligned with the repository's baseline engineering standards.
• Preserve compatibility with existing project conventions unless the skill explicitly requires a stronger standard.
• Prefer deterministic, reviewable steps over vague advice or tool-specific magic.

Anti-Patterns

• Treating examples as copy-paste truth without checking fit, constraints, or failure modes.
• Loading every reference file by default instead of using progressive disclosure.

Outputs

• A concrete result that fits the task: implementation guidance, review findings, architecture decisions, templates, or generated artifacts.
• Clear assumptions, tradeoffs, or unresolved gaps when the task cannot be completed from available context alone.
• References used, companion skills, or follow-up actions when they materially improve execution.

Evidence Produced

Category	Artifact	Format	Example
Correctness	Model evaluation report	Markdown doc covering train/test split, baseline comparison, and per-segment metrics	docs/python/ml-eval-2026-04-16.md
Operability	Model deployment runbook	Markdown doc per skill-composition-standards/references/runbook-template.md covering deploy, drift detection, and re-train procedure	docs/python/ml-runbook.md

References

• Use the references/ directory for deep detail after reading the core workflow below.

Real statistical / machine-learning models for SaaS features: forecasting, classification, regression, anomaly detection. Complements the LLM-based ai-* skills — use ML/stats when the problem is numeric, data-rich, and requires explainable, stable outputs.

Prerequisites: Load python-modern-standards and python-saas-integration before this skill. Load python-data-analytics for feature engineering.

When this skill applies

• Demand / sales / cash-flow forecasting (time-series).
• Churn prediction, credit / risk scoring, fraud flagging (classification).
• Price optimization, quantity regression (regression).
• Anomaly detection on SaaS metrics (transaction volume, latency, error rates).
• Any feature where a PHP developer would write a rule that needs tuning from data.

When ML vs LLM vs rules (decision rule)

Output is a number or label derived from many numeric features  -> ML
Output is text, summary, classification of language, extraction -> LLM
Output can be stated in < 5 unambiguous rules                    -> Rules (PHP)
Hybrid (e.g., LLM extracts features -> ML classifies)            -> both, pipeline

Rules beat ML whenever you can enumerate them — faster, explainable, testable. Reach for ML when the signal is in the data, the rules are fuzzy, and the cost of a wrong answer is modest.

See references/when-ml-vs-llm-vs-rules.md.

Core stack

• scikit-learn — classification, regression, clustering, model pipelines, metrics. Default for non-timeseries.
• statsmodels — ARIMA / ETS / SARIMA, rigorous statistical output.
• Prophet (or NeuralProphet) — seasonal + holiday-aware forecasts with minimal tuning.
• XGBoost / LightGBM — when trees beat linear models, which is often.
• PyOD — anomaly detection algorithms (isolation forest, LOF, ECOD).
• SHAP — explainability.
• joblib — model serialization.
• numpy / pandas — pre-/post-processing.

Avoid starting with deep learning. Tabular SaaS problems are almost always best solved with gradient-boosted trees or linear models. Deep learning earns its keep only for unstructured data (text, images, audio) — for which we usually call LLMs or pre-trained models instead of training from scratch.

Data preparation discipline

Leakage is the silent killer. Prevent it with process, not vigilance.

Splits:

• Random split for IID tabular → train_test_split(..., stratify=y).
• Time-series → time-based split only. Training data must precede validation, which must precede test. No shuffling.
• Grouped data (per customer, per tenant) → GroupKFold. Never let the same customer appear in train and test.

Leakage sources to eliminate:

• Target-derived features (e.g., "days since last invoice" when predicting next invoice).
• Future data in features (aggregations that include the label row).
• Preprocessing on full data before split (fit scalers/encoders on train only).

Always wrap preprocessing in a Pipeline so fit-on-train/apply-on-test is automatic:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier

preprocess = ColumnTransformer([
    ("num", StandardScaler(), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore", min_frequency=10), categorical_cols),
])

model = Pipeline([("prep", preprocess), ("clf", GradientBoostingClassifier(random_state=42))])
model.fit(X_train, y_train)   # scaler fits only on train

See references/data-prep.md.

Time-series forecasting

Prophet — use when you have daily or weekly data with yearly / weekly seasonality and holidays matter. Minimal tuning, business-friendly confidence intervals.

from prophet import Prophet

df_fc = df.rename(columns={"date": "ds", "sales": "y"})
m = Prophet(yearly_seasonality=True, weekly_seasonality=True, holidays=kenya_holidays)
m.fit(df_fc)
future = m.make_future_dataframe(periods=90, freq="D")
forecast = m.predict(future)     # yhat, yhat_lower, yhat_upper

statsmodels ARIMA / SARIMA / ETS — use when you need rigor (AIC, residual diagnostics, significance), simple series without holidays, or when Prophet is overkill.

from statsmodels.tsa.statespace.sarimax import SARIMAX

model = SARIMAX(y_train, order=(1, 1, 1), seasonal_order=(1, 1, 1, 12))
res = model.fit(disp=False)
forecast = res.get_forecast(steps=12)
mean = forecast.predicted_mean
ci = forecast.conf_int()

Always produce intervals, not just point forecasts. Users need to see uncertainty. See references/forecasting-prophet.md and references/forecasting-statsmodels.md.

Classification & regression (tabular)

Default model order to try:

1. Logistic regression (classification) / Ridge (regression) — fast, interpretable baseline.
2. Gradient Boosting (sklearn / XGBoost / LightGBM) — usually the winner on tabular.
3. Random Forest — second try if GB is unstable.
4. Neural network — only if you have > 100K rows, heavy feature engineering fails, and you have time.

Hyperparameter tuning: RandomizedSearchCV with a small budget first, then refine. Don't grid-search over 10,000 combinations.

Metrics — never just accuracy:

• Binary classification, balanced: ROC-AUC, F1.
• Binary classification, imbalanced (churn, fraud): precision@k, recall@k, precision-recall AUC.
• Multiclass: macro-F1, log-loss.
• Regression: MAE, RMSE, R². MAPE only when no zeros in target.

See references/classification-regression-sklearn.md and references/evaluation-metrics.md.

Anomaly detection

Three tiers in order of complexity:

1. Threshold on a metric (simplest): rolling mean ± 3 × rolling std. Catch big, easy anomalies, run cheap.
2. Statistical distribution: Modified Z-score (robust to outliers), ESD test, STL decomposition residuals.
3. Model-based: IsolationForest (sklearn) or ECOD (PyOD). Best for multi-feature anomalies.

from sklearn.ensemble import IsolationForest
iso = IsolationForest(contamination=0.01, random_state=42).fit(X_train)
scores = -iso.score_samples(X_new)   # higher = more anomalous

Calibrate thresholds on a known-clean period. Re-calibrate monthly. See references/anomaly-detection.md.

Model serving

Lightweight by default. No MLflow/Kubeflow unless you're genuinely running dozens of models.

Serialization: joblib.dump(model, path). Pin sklearn/xgboost versions in pyproject.toml — pickle format breaks across major versions.

Loading: eager-load at sidecar startup or worker startup. Keep the model in memory for the life of the process. Never load per request.

# src/service_name/ml/churn.py
from joblib import load
from pathlib import Path

MODEL_PATH = Path(__file__).parent / "artifacts" / "churn_v3.joblib"
_model = load(MODEL_PATH)     # loads once at import

def score(features: dict) -> float:
    X = build_feature_frame(features)
    return float(_model.predict_proba(X)[0, 1])

Versioning: filename includes semver or date (churn_v3.joblib, demand_20260301.joblib). Current version is a symlink. Rollback = flip symlink + restart.

A/B tests / shadow mode: run new model alongside old on same inputs, log both predictions, compare offline.

See references/model-serving.md.

Monitoring & drift

Models degrade silently. Detect it.

Feature drift: distributions of inputs shift over time. Monitor mean, std, quantiles of each input feature vs. a reference window. Use population stability index (PSI) or Kolmogorov-Smirnov statistic.

Prediction drift: distribution of predictions shifts. Cheap proxy for feature drift.

Performance drift: requires labels (arriving later). Log predictions, match to truth when it arrives, compute rolling metric.

Alert when: PSI > 0.2 for any feature, or monthly metric deviates > N% from training baseline.

Retraining triggers: calendar (monthly/quarterly), drift alert, or drop in business KPI.

See references/monitoring-and-drift.md.

Explainability

Users and regulators ask "why?" Have an answer.

Global: feature importance from the model (model.feature_importances_ for trees, coefficients for linear).

Local (per prediction): SHAP values. Log top-3 contributing features per prediction for high-stakes scores (credit, risk, fraud).

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_sample)

Confidence intervals on predictions matter as much as the prediction itself. Regressors: use quantile regressors or bootstrap. Classifiers: calibrate probabilities with CalibratedClassifierCV.

See references/explainability.md.

Integration patterns

Sidecar (sync scoring): small, fast models. POST /score → features in → prediction + confidence out. Latency budget < 200ms.

Worker (batch scoring): nightly scoring of all customers for churn; bulk forecast for all SKUs. Writes results to MySQL where PHP reads them.

Training: worker job, runs on a schedule. Output: new model artifact + metrics report. Never train in a sidecar.

Pitfalls specific to SaaS

• Multi-tenant models: one global model vs. per-tenant models? Start global + tenant as a feature. Go per-tenant only with evidence.
• Cold start: new tenants have no history. Fall back to global model or rules. Plan this from day one.
• Label leakage via user actions: if users can act on a prediction (e.g., flagging fraud), make sure those actions don't become features for the next model version without an offset.
• Class imbalance: 2% churn rate = baseline accuracy of 98% with a constant "no" predictor. Use class_weight="balanced", or downsample, or use threshold tuning.
• Currency / units: check whether you're predicting log(amount) or amount; mismatched exponents produce nonsense.

Anti-patterns

• Training on the full dataset "to get better results." You can't measure the result.
• Deploying a model with no baseline (constant predictor, simple rule). You won't know if it's actually helping.
• Serving with model.predict(X) inside the request handler without timeout or concurrency limits.
• Storing model artifacts in git. Use a models/ directory that's gitignored; artifact store is S3 or similar.
• Retraining nightly "just because." Retrain on a trigger.
• Ignoring calibration on probability outputs shown to users. A "40% churn risk" is meaningless unless calibrated.

References

• references/when-ml-vs-llm-vs-rules.md
• references/data-prep.md
• references/forecasting-prophet.md
• references/forecasting-statsmodels.md
• references/classification-regression-sklearn.md
• references/anomaly-detection.md
• references/evaluation-metrics.md
• references/model-serving.md
• references/monitoring-and-drift.md
• references/explainability.md

See also

• ai-predictive-analytics — LLM-based prediction (use when features are unstructured text).
• ai-evaluation — for LLM output quality; ML evaluation is different (this skill).
• saas-business-metrics — to decide which outcome to model.