data-ingestion

name	data-ingestion
description	Design data ingestion into landing/bronze layers. Use for source onboarding, transfer pattern selection, full/incremental/CDC scope, change detection, landing zones, schema drift, idempotent file or batch processing, and ingestion quality gates.

Data Ingestion

Use this skill to choose and verify an ingestion design. Keep source-specific code in the project, not in the skill.

Required Decisions

• Source type: database, API, file drop, stream, SaaS connector, data share.
• Transfer pattern: file transfer, direct query, platform connector, streaming, replication, push/drop.
• Scope: full refresh, incremental append, incremental merge, CDC with deletes.
• Change detection: watermark, source modified timestamp, sequence, CDC log, file manifest, snapshot diff.
• Landing contract: raw preservation, bronze schema, metadata columns, quarantine path.
• Idempotency: replay key, processed-file tracking, MERGE/upsert, partition overwrite, checkpoint.
• Schema evolution: additive handling, breaking-change gate, drift alert, consumer impact.
• Quality gates: row count, schema, freshness, uniqueness/dedup, reconciliation.

Selection Rules

• Prefer platform-native connectors only when they expose observability, retry behavior, and schema controls.
• Prefer file landing when source extraction and downstream processing need independent retries.
• Use incremental merge when source records can change after first arrival.
• Use CDC when deletes or update history matter.
• Use full refresh only for small data, immutable snapshots, or sources without reliable change signals.
• For streams, define checkpoint, watermark, late-data policy, replay window, and exactly-once sink behavior.

Anti-Patterns

• Inferring production schemas.
• Treating ingestion success as row movement only; quality gates still required.
• Reprocessing files without a manifest or idempotent sink.
• Overwriting whole tables when partition or key-level merge is enough.
• Allowing source schema breaks to silently flow into silver/gold.
• Landing only transformed data with no raw/audit trail.

Verification

Before marking ingestion design or code done:

• Re-run same batch/window and prove no duplicates.
• Compare source and target counts for the chosen scope.
• Validate schema at source boundary and before write.
• Simulate late, duplicate, and missing source records.
• Confirm quarantine or fail-fast behavior for bad records.
• Confirm monitoring includes freshness, volume, failures, and drift.