القائمة
Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
| name | data-deduplicator |
| description | Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies. |
| version | 0.5.0 |
Detects and removes duplicate records in tabular data. Supports exact match on all columns, exact match on a subset of key columns (composite key), and fuzzy matching for near-duplicate detection. Reports all duplicates found before removal, keeping configurable occurrence (first, last, or none).
--input / -i: Path to input CSV or Parquet (required)--mode: Deduplication mode (default: exact-all)--key-columns / -k: Columns for composite key matching (comma-separated)--keep: Which duplicate to keep: first, last, or none (default: first)--threshold: Fuzzy matching similarity threshold 0-100 (default: 90)--output / -o: Output path for deduplicated data--report-only: Only report duplicates without removingkeep strategyduplicates_report.json: Duplicate groups with row indices, count of removed rowspython3 ./skills/data_deduplicator/dedup.py -i <file> -o <output> --mode <mode>pandas, fuzzywuzzy (for fuzzy mode)Validate tabular data against configurable business rules — range checks, cross-column constraints, referential integrity against lookup tables, and temporal consistency.
Load and parse CSV files into structured data frames with configurable encoding, delimiter, and date parsing options.
Export pandas DataFrames or processed datasets to well-formatted CSV files with configurable encoding, delimiter, quoting, and column ordering.
Aggregate tabular data by one or more dimensions with configurable metrics — sum, mean, count, min, max, median. Supports multi-level grouping and pivot table generation.
Comprehensive data cleaning toolkit — handles missing values, duplicate removal, type coercion, whitespace trimming, and outlier detection in a single pass over the dataset.
Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies.