القائمة
Universal data ingestion — loads CSV, Excel, JSON, and Parquet files into unified DataFrames with auto-detection of format, encoding, and date columns. Supports glob patterns for batch loading.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
Validate tabular data against configurable business rules — range checks, cross-column constraints, referential integrity against lookup tables, and temporal consistency.
Load and parse CSV files into structured data frames with configurable encoding, delimiter, and date parsing options.
Export pandas DataFrames or processed datasets to well-formatted CSV files with configurable encoding, delimiter, quoting, and column ordering.
Aggregate tabular data by one or more dimensions with configurable metrics — sum, mean, count, min, max, median. Supports multi-level grouping and pivot table generation.
Comprehensive data cleaning toolkit — handles missing values, duplicate removal, type coercion, whitespace trimming, and outlier detection in a single pass over the dataset.
Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies.
| name | data-ingestion |
| description | Universal data ingestion — loads CSV, Excel, JSON, and Parquet files into unified DataFrames with auto-detection of format, encoding, and date columns. Supports glob patterns for batch loading. |
| version | 1.0.0 |
Consolidated data loading service that ingests data from all common tabular formats (CSV, Excel/xlsx/xls, JSON/JSONL, Parquet) and normalizes them into pandas DataFrames. Auto-detects file format by extension, handles encoding detection (chardet), parses date columns, strips whitespace from string fields, and supports batch ingestion via glob patterns.
Merges functionality from: csv_data_loader, excel_data_reader, data_ingestion_service.
.csv, .tsv, .txt): Configurable delimiter, encoding, quoting.xlsx, .xls): Multi-sheet support via openpyxl/xlrd, header row selection.json, .jsonl): Nested object flattening, JSON Lines support.parquet): Direct columnar read with partition support--input / -i: Path or glob pattern for input files (required)--format / -f: Force file format (auto-detected if omitted)--encoding / -e: File encoding for text formats (default: auto-detect via chardet)--delimiter / -d: Delimiter for CSV files (default: ,)--sheet: Sheet name/index for Excel files (default: first sheet; all for all sheets)--date-columns: Columns to parse as datetime--output / -o: Output directory--format override)python3 ./skills/data_ingestion/ingest.py -i <pattern> -o <output_dir>pandas, openpyxl, xlrd, chardet, pyarrow