메뉴
Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Validate tabular data against configurable business rules — range checks, cross-column constraints, referential integrity against lookup tables, and temporal consistency.
Load and parse CSV files into structured data frames with configurable encoding, delimiter, and date parsing options.
Export pandas DataFrames or processed datasets to well-formatted CSV files with configurable encoding, delimiter, quoting, and column ordering.
Aggregate tabular data by one or more dimensions with configurable metrics — sum, mean, count, min, max, median. Supports multi-level grouping and pivot table generation.
Comprehensive data cleaning toolkit — handles missing values, duplicate removal, type coercion, whitespace trimming, and outlier detection in a single pass over the dataset.
Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies.
| name | data-deduplicator |
| description | Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies. |
| version | 0.5.0 |
Detects and removes duplicate records in tabular data. Supports exact match on all columns, exact match on a subset of key columns (composite key), and fuzzy matching for near-duplicate detection. Reports all duplicates found before removal, keeping configurable occurrence (first, last, or none).
--input / -i: Path to input CSV or Parquet (required)--mode: Deduplication mode (default: exact-all)--key-columns / -k: Columns for composite key matching (comma-separated)--keep: Which duplicate to keep: first, last, or none (default: first)--threshold: Fuzzy matching similarity threshold 0-100 (default: 90)--output / -o: Output path for deduplicated data--report-only: Only report duplicates without removingkeep strategyduplicates_report.json: Duplicate groups with row indices, count of removed rowspython3 ./skills/data_deduplicator/dedup.py -i <file> -o <output> --mode <mode>pandas, fuzzywuzzy (for fuzzy mode)