data-mining

name

data-mining

description

Expert data mining workflow for tabular datasets — covers dataset discovery, quality verification, exploratory data analysis, and predictive modeling in Python. Use this skill whenever the user mentions data mining, EDA, exploratory analysis, dataset analysis, data quality checks, building predictive models from data, finding datasets, or any task that involves loading a CSV/Excel file and extracting insights. Also trigger when the user asks about Kaggle datasets, UCI ML repository, or wants help choosing a dataset for a project. Even if the user just says "analyze this data" or "what can I learn from this file," this skill provides the structured workflow to follow.

Data Mining Expert

Generate a single Jupyter notebook (.ipynb) the user can run and reproduce. Use pandas, numpy, matplotlib, seaborn, scikit-learn, scipy. Add %matplotlib inline at the top.

What separates this from a generic analysis: leakage prevention, proper validation, domain interpretation, and honest limitations.

If you need help finding a dataset, read references/dataset_search.md. Otherwise, all guidance is below — no additional reference files needed.

Phase 1: Data Quality

Check in this order:

1. Data leakage — Look for columns that encode the target (r > 0.9 with target), are only knowable after prediction, or are proxies/IDs. Report and drop them before modeling. After modeling, check if R² > 0.99 or accuracy > 0.99 — if so, investigate leakage.

2. Missing values — Count per column. When imputing, add a col_was_missing binary indicator — the fact that data is missing is often predictive.

3. Outliers — IQR method. Flag but don't remove. Distinguish "measurement error" from "genuine extreme value."

4. Types, duplicates, constant columns — Quick sanity check.

Phase 2: EDA

Every chart gets a one-sentence takeaway. Without it, a plot is decoration.

Cover:

• Distributions — Note heavy skewness (|skew| > 1), suggest log1p transform
• Target analysis — If classification, check class balance. If imbalance ratio > 3:1, plan for class_weight='balanced' and stratified splitting. If regression, check for capped/truncated target values.
• Correlations — Flag feature pairs with |r| > 0.85 as redundant (multicollinearity). Show the correlation matrix.
• Feature vs target — Top 5-6 features by correlation, scatter plots with trend lines for regression, distribution overlays per class for classification
• Pair plot for top features (sample to 2000 rows if dataset is large)

Use domain language: "body mass index (BMI)" not "feature_2".

Phase 3: Modeling

Use scikit-learn Pipeline + ColumnTransformer for all preprocessing. This prevents data leakage from imputation statistics bleeding across train/test.

1. Split BEFORE fitting anything. Stratified for classification.
2. Numeric: median impute → StandardScaler. Categorical: mode impute → OneHotEncoder.
3. Feature selection — Reduce dimensionality inside the Pipeline to prevent overfitting and speed up training. Apply in order:
   • VarianceThreshold — drop near-constant features (threshold ≈ 0.01).
   • Drop one of any feature pair with |r| > 0.85 (the one less correlated with target) to remove multicollinearity flagged in EDA.
   • SelectKBest(f_classif or f_regression) or mutual_info_classif/reg — keep the top K informative features (K ≈ min(20, n_features)).
   • Optional: RFE or SelectFromModel with a L1-regularized model for a second pass if dimensionality is still high (>50 features).
   • Report how many features were kept vs. dropped, and list the dropped features with brief rationale.
4. Compare 3+ algorithms (include at least one linear and one tree-based).
5. Cross-validate (stratified k-fold for classification). Report multiple metrics — accuracy alone is never enough.
6. Tune the best model with RandomizedSearchCV (small grid: 3-5 params, 3-5 values).
7. Interpret — Feature importances + permutation importance as sanity check. Explain top features in domain terms.
8. Error analysis — Show where the model fails (worst predictions, confusion matrix, residual distribution).
9. Limitations — End with an explicit section on what the model can't do, sample size concerns, and features that might not generalize.

Notebook structure

[Markdown] # Data Mining Analysis: {dataset_name}
[Code]     # Imports (%matplotlib inline)
[Markdown] ## Data Quality
[Code]     # Load + quality checks + leakage detection
[Markdown] ## Exploratory Data Analysis
[Code]     # Visualizations with interpretations
[Markdown] ## Modeling Pipeline
[Code]     # Pipeline, model comparison, tuning, feature importance, error analysis
[Markdown] ## Conclusions & Limitations