datasource-connectors

Ingestion patterns for CSV, JSON, REST API, SQL, Excel, and Parquet -- guides an LLM through loading data from any common source

Zur Installation springen

Repository: fabioc-aloha/Alex_ACT_Visual_Storytelling
Letzte Quellaktivität: 16. August 2026 um 01:10
Erkannte Sprache von SKILL.md: Englisch
Sterne: 0
Forks: 0

Installationsoptionen

Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.

Quelldateien prüfen

Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.

Zurück zur Suche Zurück zur Installation

type	skill
lifecycle	stable
inheritance	inheritable
name	datasource-connectors
description	Ingestion patterns for CSV, JSON, REST API, SQL, Excel, and Parquet -- guides an LLM through loading data from any common source
tier	standard
applyTo	*/data,/ingest,/connect,/csv,/json,/sql,/excel,/parquet,/api*
currency	2026-08-15T00:00:00.000Z
lastReviewed	2026-08-15T00:00:00.000Z

Datasource Connectors

Guide the user through loading data from any common source into a working dataset. This module is a decision framework, not a library. It tells the LLM what to ask, what to watch for, and how to handle each format's quirks.

When to Use

The brief names a data source that is not yet loaded
The user says "the data is in X" where X is a file, database, or API
The orchestrator's plan includes an Ingest step

When to Skip

Data is already loaded (inline table, prior step output, user pasted it)
The brief says "data is pre-loaded" or "use this dataframe"

Connector Selection

Ask the user what format the data is in, or infer from the path/URL:

Signal	Connector
`.csv`, `.tsv`, `.txt` (tabular)	CSV
`.json`, `.jsonl`, `.ndjson`	JSON
`http://`, `https://` + returns JSON/XML	REST API
Connection string, `.sql`, database name	SQL
`.xlsx`, `.xls`	Excel
`.parquet`, `.arrow`	Parquet

If ambiguous, ask. Do not guess the format.

CSV / TSV

What to ask

Does the file have a header row? (default: yes)
What is the delimiter? (auto-detect: comma, tab, semicolon, pipe)
What is the encoding? (default: UTF-8; watch for: Latin-1, Windows-1252)

Quirks to handle

Problem	Detection	Fix
Wrong delimiter	First row has one column	Try tab, semicolon, pipe
Encoding garbled	Non-ASCII chars show as `?` or `Ã©`	Re-read as Latin-1 or Windows-1252
Trailing commas	Row has one extra empty column	Strip trailing delimiter

Problem	Detection	Fix
Nested objects	Column values are objects, not scalars	Flatten with dot notation (`address.city`)
Mixed types in array	Some items have extra/missing keys	Union all keys; fill missing with null
Large file (> 100MB)	Slow to parse	Stream with JSONL or chunk
Date strings	ISO 8601 or Unix timestamps	Parse to date; state the format detected

Method	How to use	Security
API key in header	`Authorization: Api-Key <key>` or custom header	Never log the key
Bearer token	`Authorization: Bearer <token>`	Store in env var or secret
OAuth 2.0	Client credentials or authorization code flow	Use refresh tokens
No auth	Public API	Still rate-limit respectfully

Pattern	Detection	Handling
Offset	`?offset=0&limit=100`	Increment offset until empty page
Cursor	Response has `next_cursor` field	Pass cursor to next request
Link header	`Link: <url>; rel="next"`	Follow `rel="next"` until absent
Page number	`?page=1&per_page=50`	Increment page until empty

Rule	Reason
Always use parameterized queries	Prevents SQL injection
Never interpolate user input into SQL	Even for table names; use allowlists
Read-only connection	The pipeline only reads; never write
Credentials in env vars	Never in code, never in logs

-- Simple table read
SELECT * FROM schema.table_name LIMIT 1000;

-- Filtered read (parameterized)
SELECT * FROM sales WHERE region = @region AND date >= @start_date;

-- Aggregated read (push aggregation to the database when possible)
SELECT region, month, SUM(revenue) AS revenue
FROM sales
GROUP BY region, month;

Problem	Detection	Fix
Multiple sheets	Workbook has > 1 sheet	Ask which sheet; default to first
Header not in row 1	First rows are title/metadata	Ask for header row number
Merged cells	Values span multiple rows/columns	Unmerge and fill down
Formulas	Cells contain formulas, not values	Read values, not formulas
Mixed types in column	Numbers and text in same column	Coerce to string; flag for data-preparation
Date serial numbers	Dates stored as integers (e.g., 45678)	Convert using Excel epoch (1899-12-30)

Problem	Detection	Fix
Partitioned directory	Path is a directory with `part-*.parquet` files	Read all partitions
Nested columns (struct/list)	Schema shows complex types	Flatten or ask user which fields
Large file	> 1M rows	Sample or filter before loading all

Situation	Action
File not found	Report the exact path tried. Ask user to verify.
Permission denied	Report the error. Do not retry. Ask user to fix permissions.
Empty dataset (0 rows)	Report it. Push back: "file exists but has no data rows."
Partial data (truncated)	Report row count. Ask if this is expected.
Unsupported format	State what was detected. Ask user for the correct format.