القائمة
Universal data ingestion service that loads data from CSV, Excel, JSON, and Parquet sources into a unified DataFrame format with automatic format detection.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
Validate tabular data against configurable business rules — range checks, cross-column constraints, referential integrity against lookup tables, and temporal consistency.
Load and parse CSV files into structured data frames with configurable encoding, delimiter, and date parsing options.
Export pandas DataFrames or processed datasets to well-formatted CSV files with configurable encoding, delimiter, quoting, and column ordering.
Aggregate tabular data by one or more dimensions with configurable metrics — sum, mean, count, min, max, median. Supports multi-level grouping and pivot table generation.
Comprehensive data cleaning toolkit — handles missing values, duplicate removal, type coercion, whitespace trimming, and outlier detection in a single pass over the dataset.
Identify and remove duplicate records from tabular datasets using exact-match, fuzzy-match, or composite-key-based deduplication strategies.
| name | data-ingestion-service |
| description | Universal data ingestion service that loads data from CSV, Excel, JSON, and Parquet sources into a unified DataFrame format with automatic format detection. |
| version | 1.1.0 |
A universal data loading service that ingests data from multiple file formats (CSV, Excel, JSON, Parquet) and normalizes them into a unified pandas DataFrame representation. Automatically detects file format by extension, applies format-specific parsing options, handles encoding detection, and supports batch ingestion via glob patterns.
.csv, .tsv, .txt): Configurable delimiter, encoding, quoting.xlsx, .xls): Multi-sheet support, header row selection.json, .jsonl): Nested object flattening, JSON Lines support.parquet): Direct columnar read, partition support--input / -i: Path or glob pattern for input files (required)--format / -f: Force file format (auto-detected if omitted)--encoding / -e: File encoding for text formats (default: auto-detect)--delimiter / -d: Delimiter for CSV files (default: ,)--sheet: Sheet name for Excel files (default: first sheet)--date-columns: Columns to parse as dates--output / -o: Output directory--format override)python3 ./skills/data_ingestion_service/ingest.py -i <pattern> -o <output_dir>pandas, openpyxl, chardet, pyarrow