菜单
Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Validate tabular data against configurable business rules — range checks, cross-column constraints, referential integrity against lookup tables, and temporal consistency.
Load and parse CSV files into structured data frames with configurable encoding, delimiter, and date parsing options.
Export pandas DataFrames or processed datasets to well-formatted CSV files with configurable encoding, delimiter, quoting, and column ordering.
Aggregate tabular data by one or more dimensions with configurable metrics — sum, mean, count, min, max, median. Supports multi-level grouping and pivot table generation.
Comprehensive data cleaning toolkit — handles missing values, duplicate removal, type coercion, whitespace trimming, and outlier detection in a single pass over the dataset.
Universal data ingestion service that loads data from CSV, Excel, JSON, and Parquet sources into a unified DataFrame format with automatic format detection.
| name | data-deduplicator |
| description | Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies. |
| version | 0.5.0 |
Detects and removes duplicate records in tabular data. Supports exact match on all columns, exact match on a subset of key columns (composite key), and fuzzy matching for near-duplicate detection. Reports all duplicates found before removal, keeping configurable occurrence (first, last, or none).
--input / -i: Path to input CSV or Parquet (required)--mode: Deduplication mode (default: exact-all)--key-columns / -k: Columns for composite key matching (comma-separated)--keep: Which duplicate to keep: first, last, or none (default: first)--threshold: Fuzzy matching similarity threshold 0-100 (default: 90)--output / -o: Output path for deduplicated data--report-only: Only report duplicates without removingkeep strategyduplicates_report.json: Duplicate groups with row indices, count of removed rowspython3 ./skills/data_deduplicator/dedup.py -i <file> -o <output> --mode <mode>pandas, fuzzywuzzy (for fuzzy mode)