菜单
Provides string manipulation and text analysis functions for DataFrames. Use when the user asks to one-hot encode comma-separated tags into boolean columns, or needs to extract statistically significant terms from a subset of text documents in a Python data science project.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Provides data preprocessing visualization and statistical correlation methods. Use when the user asks to visualize feature distributions, wants to plot correlation matrices or dendrograms, explore feature interactions, or needs to calculate mutual information or label-based statistics in a Python data science project using sklearn-compatible models.
Provides evaluation metrics and visualization charts for machine learning models. Use when the user asks to evaluate a model, wants to plot a confusion matrix, ROC curve, or Precision-Recall curve, needs to analyze learning curves, probability distributions, or error analysis in a Python data science project using sklearn-compatible models.
Sklearn pipeline transformers for multi-label binarization and sentence embeddings. Use when the user needs MultiLabelBinarizer in a Pipeline, ColumnTransformer, or set_output(transform="pandas"), or when integrating sentence-transformers models into sklearn pipelines for NLP tasks.
Provides Explainable AI (XAI) tools to interpret machine learning models. Use when the user asks to visualize feature importance for tree-based models, needs to explain which features drive model decisions, wants to render or draw a decision tree diagram, or needs to convert a Graphviz DOT string into a matplotlib figure in a Python data science project using sklearn-compatible models.
Provides evaluation and visualization for unsupervised learning and clustering. Use when the user asks to plot cluster cardinality, wants to visualize cluster magnitude, compare magnitude versus cardinality to find anomalies, or needs to determine the optimal number of clusters in a Python data science project using sklearn-compatible models.
| name | ds-utils-strings |
| description | Provides string manipulation and text analysis functions for DataFrames. Use when the user asks to one-hot encode comma-separated tags into boolean columns, or needs to extract statistically significant terms from a subset of text documents in a Python data science project. |
| license | MIT |
| metadata | {"author":"Idan Morad","documentation":"https://datascienceutils.readthedocs.io/en/stable/","package":"data-science-utils","repository":"https://github.com/idanmoradarthas/DataScienceUtils"} |
String and tags manipulation tools for feature extraction.
pip install data-science-utils
# or
conda install -c idanmorad data-science-utils
from ds_utils.strings import append_tags_to_frame
from ds_utils.strings import extract_significant_terms_from_subset
Extracts tags from a specified field and creates new binary columns (One-Hot Encoded) for each unique tag.
from ds_utils.strings import append_tags_to_frame
# complete usage example
x_train_tags, x_test_tags = append_tags_to_frame(
x_train, x_test, "article_tags", prefix="tag_"
)
Parameters:
x_train — pd.DataFrame, The training dataframe.x_test — pd.DataFrame, The test dataframe.tag_column — str, The column containing comma-separated tags.prefix — str, Prefix for the generated boolean tag columns.Returns: Tuple[pd.DataFrame, pd.DataFrame] representing the one-hot encoded dataset features.
Output Example:
If x_train has a column tags with "ds,ml" for article_1 and "ml,dl" for article_2:
| article_name | tag_ds | tag_ml | tag_dl |
|---|---|---|---|
| article_1 | 1 | 1 | 0 |
| article_2 | 0 | 1 | 1 |
Common mistakes:
append_tags_to_frame MUST receive BOTH train AND test together. Calling it separately on each will produce mismatched columns and data leakage.x_train only; test-only tags are silently dropped.Identifies terms that are statistically overrepresented in a subset of documents compared to the full corpus.
from ds_utils.strings import extract_significant_terms_from_subset
# complete usage example
subset = df[df['label'] == 'interesting']
terms_scores = extract_significant_terms_from_subset(df, subset, "text_column")
Parameters:
data_frame — pd.DataFrame, The full corpus representing all documents.subset — pd.DataFrame, The subset of the dataframe representing specific documents to evaluate against the corpus.field — str, Target text or document content column.Returns: pd.Series — index contains the terms, values are significance
scores from 0.0 to 1.0. Access top terms with .sort_values(ascending=False).head(10).
Output Example: If identifying distinctive terms in the subset documents:
| index (term) | value (score) |
|---|---|
| third | 1.0 |
| one | 1.0 |
| and | 1.0 |
| this | 0.67 |
| document | 0.25 |
Access as: terms["third"] → 1.0 or terms.sort_values(ascending=False)
Common mistakes:
import pandas as pd
from ds_utils.strings import append_tags_to_frame
x_train = pd.DataFrame([{"id": 1, "tags": "ds,ml,dl"}])
x_test = pd.DataFrame([{"id": 2, "tags": "ds,ml"}])
x_train_out, x_test_out = append_tags_to_frame(x_train, x_test, "tags", "tag_")