| name | ds-utils-preprocess |
| description | Provides data preprocessing visualization and statistical correlation methods. Use when the user asks to visualize feature distributions, wants to plot correlation matrices or dendrograms, explore feature interactions, or needs to calculate mutual information or label-based statistics in a Python data science project using sklearn-compatible models.
|
| license | MIT |
| metadata | {"author":"Idan Morad","documentation":"https://datascienceutils.readthedocs.io/en/stable/","package":"data-science-utils","repository":"https://github.com/idanmoradarthas/DataScienceUtils"} |
Preprocess — ds_utils.preprocess
Utilities for data visualization and robust feature statistics.
Installation
pip install data-science-utils
conda install -c idanmorad data-science-utils
Import
from ds_utils.preprocess.visualization import visualize_feature
from ds_utils.preprocess.visualization import visualize_correlations
from ds_utils.preprocess.visualization import plot_correlation_dendrogram
from ds_utils.preprocess.visualization import plot_features_interaction
from ds_utils.preprocess.visualization import plot_pca_explained_variance
from ds_utils.preprocess.statistics import get_correlated_features
from ds_utils.preprocess.statistics import extract_statistics_dataframe_per_label
from ds_utils.preprocess.statistics import compute_mutual_information
visualize_feature
Visualizes a single feature's distribution based on its data type.
from ds_utils.preprocess.visualization import visualize_feature
visualize_feature(
df["feature_name"],
remove_na=False,
include_outliers=True,
outlier_iqr_multiplier=1.5,
first_day_of_week="Monday",
show_counts=True,
order=None
)
Parameters:
series — pd.Series, The feature to visualize.remove_na — bool, whether to explicitly remove NAs before plotting. (Default: False).include_outliers — bool, For float features, whether to include outliers in the violin plot. (Default: True).outlier_iqr_multiplier — float, IQR multiplier for outlier fence computation if include_outliers=False. (Default: 1.5).first_day_of_week — str, Start day for datetime heatmaps (e.g., "Monday" or "Sunday"). (Default: "Monday").show_counts — bool, Whether to display count labels on categorical bar charts. (Default: True).order — str or list, Custom sorting for categories (e.g. "count_desc" or ["High", "Low"]). (Default: None).
Returns: matplotlib Axes.
Common mistakes:
- Always pass a Series (e.g.,
df["col"]), never a DataFrame (df[["col"]]).
- The function auto-selects the plot by dtype (float→violin, datetime→heatmap, rest→bar). You don't need to specify the plot type.
- High-cardinality features (>10 unique values) will only show the top 10 categories, with the rest grouped into "Other values".
visualize_correlations
Visualizes a correlation matrix as a heatmap.
from ds_utils.preprocess.visualization import visualize_correlations
import matplotlib.pyplot as plt
visualize_correlations(df.corr())
plt.show()
Parameters:
correlation_matrix — pd.DataFrame, The correlation matrix.
Returns: matplotlib Axes.
Common mistakes:
- Always pass the correlation matrix
df.corr() as the input result, NOT the raw DataFrame df.
plot_correlation_dendrogram
Plots a hierarchical dendrogram of a correlation matrix.
from ds_utils.preprocess.visualization import plot_correlation_dendrogram
import matplotlib.pyplot as plt
plot_correlation_dendrogram(df.corr())
plt.show()
Parameters:
correlation_matrix — pd.DataFrame, The correlation matrix.
Returns: matplotlib Axes.
Common mistakes:
- Passing the raw DataFrame instead of
df.corr().
plot_features_interaction
Plots the joint distribution and relationship between two features.
from ds_utils.preprocess.visualization import plot_features_interaction
plot_features_interaction(df, "feature1", "feature2", remove_na=False)
Parameters:
data_frame — pd.DataFrame, The dataset.feature_1 — str, First feature column name.feature_2 — str, Second feature column name.remove_na — bool, Whether to remove NA/null values before plotting.
When False (default), missing values are included: shown as rug plots
on the axes for numeric/datetime plots, or as a separate bar for
categorical plots.
Dtype → Chart type matrix:
| feature_1 ↓ feature_2 → | Numeric | Categorical / Bool | Datetime |
|---|
| Numeric | Scatter plot | Box plot | Line plot |
| Categorical / Bool | Box plot | Shared histogram | Violin plot |
| Datetime | Line plot | Violin plot | Line plot |
Returns: matplotlib Axes.
plot_pca_explained_variance
Plots the cumulative explained variance ratio of PCA components.
from ds_utils.preprocess.visualization import plot_pca_explained_variance
plot_pca_explained_variance(df, use_scaling=True)
plot_pca_explained_variance(df, pca_kwargs={"n_components": 5})
Parameters:
X — pd.DataFrame, The dataset with numerical features.use_scaling — bool, Whether to scale the data before fitting PCA.scaler — TransformerMixin, Scaler instance to use when use_scaling is True (default: StandardScaler).legend_loc — str, Location of the legend (default: "lower right").ax — axes.Axes, Matplotlib Axes to draw the plot on.pca_kwargs — dict, Additional keyword arguments passed directly to sklearn.decomposition.PCA.
Returns: matplotlib Axes.
Common mistakes:
- Passing a DataFrame with non-numeric columns.
get_correlated_features
Extracts pairs of highly correlated features and their correlation to the target.
from ds_utils.preprocess.statistics import get_correlated_features
correlations = get_correlated_features(df.corr(), ["f1", "f2"], "target")
Parameters:
correlation_matrix — pd.DataFrame, The correlation matrix.features — list, List of feature names to consider.target — str, The target column name (to report their correlation to the target).
Returns: pd.DataFrame with columns ['level_0', 'level_1', 'level_0_level_1_corr', 'level_0_target_corr', 'level_1_target_corr'].
Output Example:
| level_0 | level_1 | level_0_level_1_corr | level_0_target_corr | level_1_target_corr |
|---|
| featureA | featureB | 0.95 | 0.51 | 0.48 |
Common mistakes:
target must be a column name string, not an array of values.correlation_matrix must be pre-computed.
extract_statistics_dataframe_per_label
Calculates statistical metrics for a feature grouped by label values.
from ds_utils.preprocess.statistics import extract_statistics_dataframe_per_label
stats_df = extract_statistics_dataframe_per_label(df, "amount", "category")
Parameters:
df — pd.DataFrame, The input data.feature_name — str, the numerical feature to describe.label_name — str, the categorical label column.
Returns: pd.DataFrame of statistics per group, containing count, mean, min, max and percentiles.
Output Example:
| category | count | null_count | mean | min | 1_percentile | 5_percentile | 25_percentile | median | 75_percentile | 95_percentile | 99_percentile | max |
|---|
| Group_A | 100 | 0 | 25.5 | 10 | 11.2 | 12.0 | 15.0 | 25.0 | 35.0 | 45.0 | 49.0 | 50 |
| Group_B | 150 | 2 | 45.0 | 20 | 21.0 | 22.0 | 30.0 | 45.0 | 60.0 | 75.0 | 80.0 | 85 |
compute_mutual_information
Computes mutual information scores between features and a target label.
from ds_utils.preprocess.statistics import compute_mutual_information
mi_scores = compute_mutual_information(df, ["f1", "f2"], "target", random_state=42)
Parameters:
df — pd.DataFrame, The data frame.features — list of str, Features to test.label_col — str, Target column.random_state — int, random state for reproducibility.
Returns: pd.DataFrame of MI scores sorted descending.
Output Example:
| feature_name | mi_score |
|---|
| feature1 | 0.245 |
| feature3 | 0.182 |
| feature2 | 0.091 |
Typical Workflow
from ds_utils.preprocess.visualization import visualize_feature, visualize_correlations
from ds_utils.preprocess.statistics import get_correlated_features
visualize_feature(df["age"])
corr = df.corr()
visualize_correlations(corr)
results = get_correlated_features(corr, ["age", "income"], "is_churn")
plot_pca_explained_variance(df[["age", "income", "balance"]])
