Use this skill when building data pipelines, ETL/ELT workflows, or data transformation layers. Triggers on Airflow DAG design, dbt model creation, Spark job optimization, streaming vs batch architecture decisions, data ingestion, data quality checks, pipeline orchestration, incremental loads, CDC (change data capture), schema evolution, and data warehouse modeling. Acts as a senior data engineer advisor for building reliable, scalable data infrastructure.
Use this skill when building data pipelines, ETL/ELT workflows, or data transformation layers. Triggers on Airflow DAG design, dbt model creation, Spark job optimization, streaming vs batch architecture decisions, data ingestion, data quality checks, pipeline orchestration, incremental loads, CDC (change data capture), schema evolution, and data warehouse modeling. Acts as a senior data engineer advisor for building reliable, scalable data infrastructure.
When this skill is activated, always start your first response with the 🧢 emoji.
Data Pipelines
A senior data engineer's decision-making framework for building production data
pipelines. This skill covers the five pillars of data engineering - ingestion
patterns (ETL vs ELT), orchestration (Airflow), transformation (dbt), large-scale
processing (Spark), and architecture choices (streaming vs batch) - with emphasis
on when to use each pattern and the trade-offs involved. Designed for engineers
who need opinionated guidance on building reliable, observable, and maintainable
data infrastructure.
When to use this skill
Trigger this skill when the user:
Designs an ETL or ELT pipeline from scratch
Writes or debugs an Airflow DAG
Creates dbt models, tests, or macros
Optimizes a Spark job (shuffles, partitioning, memory tuning)
Decides between streaming and batch processing
Implements incremental loads or change data capture (CDC)
Plans a data warehouse or lakehouse architecture
Needs data quality checks, schema evolution, or pipeline monitoring
Do NOT trigger this skill for:
BI/analytics dashboard design or visualization (use an analytics skill)
ML model training or feature engineering (use an ML/data-science skill)
Key principles
Idempotency is non-negotiable - Every pipeline run with the same input must
produce the same output. Design for safe re-runs from day one. Use date
partitions, merge keys, or upsert logic so that retries never corrupt data.
Prefer ELT over ETL in modern stacks - Load raw data first, transform in
the warehouse. This preserves the source of truth, enables schema-on-read, and
lets analysts iterate on transformations without re-ingesting. ETL still wins
when you need to filter sensitive data before it lands.
Partition and increment, never full-reload - Full table scans on every run
do not scale. Use incremental models (dbt), date-partitioned loads, and
watermarks to process only what changed. Fall back to full reload only for small
reference tables or disaster recovery.
Orchestrate, don't script - A cron job calling a Python script is not a
pipeline. Use a proper orchestrator (Airflow, Dagster, Prefect) for retries,
dependency management, backfills, and observability. The orchestrator should
own scheduling and state, not your application code.
Test data like code - Schema tests, row count checks, uniqueness
constraints, and freshness SLAs are not optional. dbt tests, Great Expectations,
or custom assertions should gate every pipeline stage. Bad data downstream is
more expensive than a failed pipeline.
Core concepts
Data pipelines move data from sources (databases, APIs, event streams) through
transformations to destinations (warehouses, lakes, serving layers). The two
dominant patterns are ETL (extract-transform-load) and ELT
(extract-load-transform). ETL transforms data in-flight before loading; ELT
loads raw data first and transforms inside the destination.
The pipeline lifecycle has four stages: ingestion (getting data in),
orchestration (scheduling and dependency management), transformation
(cleaning, joining, aggregating), and serving (making data available to
consumers). Each stage has specialized tools: Fivetran/Airbyte for ingestion,
Airflow/Dagster for orchestration, dbt for transformation, and the warehouse
itself (BigQuery, Snowflake, Redshift) for serving.
Streaming vs batch is an architecture decision, not a tool choice. Batch
processes data in time-windowed chunks (hourly, daily). Streaming processes
events continuously as they arrive. Most organizations need both - batch for
historical aggregations and streaming for real-time dashboards or alerting.
The Lambda architecture runs both in parallel; the Kappa architecture uses a
single streaming layer for everything.
Common tasks
Design an ETL/ELT pipeline
Decide the pattern based on your constraints:
Need to filter PII before landing? -> ETL (transform before load)
Want analysts to iterate on transforms? -> ELT (load raw, transform in warehouse)
Source data volume > 1TB per load? -> ELT with Spark for heavy transforms
Small reference data < 100MB? -> Direct load, skip the framework
Standard ELT flow:
Extract from source (API, database CDC, file drop)
Load raw data to staging layer (preserve original schema)
Transform in warehouse using dbt (staging -> intermediate -> mart)
Test data quality at each layer boundary
Serve from mart layer to downstream consumers
Always land raw data in an immutable staging layer. Transformations should
read from staging, never modify it. This gives you a re-playable source of truth.
Write an Airflow DAG
A well-structured DAG separates orchestration from business logic:
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from datetime import datetime, timedelta
default_args = {
"owner": "data-team",
"retries": 2,
"retry_delay": timedelta(minutes=5),
"execution_timeout": timedelta(hours=2),
}
with DAG(
dag_id="daily_orders_pipeline",
schedule="0 6 * * *",
start_date=datetime(2024, 1, 1),
catchup=False,
default_args=default_args,
tags=["production", "orders"],
) as dag:
extract = PythonOperator(
task_id="extract_orders",
python_callable=extract_orders_fn,
op_kwargs={"ds": "{{ ds }}"},
)
transform = BigQueryInsertJobOperator(
task_id="transform_orders",
configuration={"query": {"query": "{% include 'sql/transform_orders.sql' %}"}},
)
test = PythonOperator(
task_id="test_row_counts",
python_callable=assert_row_counts,
)
extract >> transform >> test
Use catchup=False for most production DAGs unless you explicitly need
backfill behavior. Set execution_timeout to prevent zombie tasks.
Build dbt models
Structure dbt projects in three layers:
models/
staging/ -- 1:1 with source tables, light renaming/casting
stg_orders.sql
stg_customers.sql
intermediate/ -- business logic joins, deduplication
int_orders_enriched.sql
marts/ -- final consumer-facing tables
fct_daily_revenue.sql
dim_customers.sql
Example incremental model:
-- models/staging/stg_orders.sql
{{
config(
materialized='incremental',
unique_key='order_id',
on_schema_change='append_new_columns'
)
}}
select
order_id,
customer_id,
order_total,
cast(created_at astimestamp) as ordered_at
from {{ source('raw', 'orders') }}
{% if is_incremental() %}
where created_at > (selectmax(ordered_at) from {{ this }})
{% endif %}
Always define unique_key for incremental models. Without it, dbt appends
instead of merging, causing duplicates on re-runs.
Optimize a Spark job
The three most common Spark performance killers and their fixes:
Problem
Symptom
Fix
Data skew
One task takes 10x longer than others
Salt the join key, or use broadcast() for small tables
Too many shuffles
High shuffle read/write in Spark UI
Repartition before joins, coalesce after filters
Small files
Thousands of tiny output files
Use repartition(N) or coalesce(N) before write
from pyspark.sql import SparkSession
from pyspark.sql.functions import broadcast
spark = SparkSession.builder.appName("optimize_example").getOrCreate()
# Broadcast small dimension table to avoid shuffle
orders = spark.read.parquet("s3://data/orders/")
products = spark.read.parquet("s3://data/products/") # < 100MB
enriched = orders.join(broadcast(products), "product_id", "left")
# Repartition by date before writing to avoid small files
enriched.repartition("order_date").write \
.partitionBy("order_date") \
.mode("overwrite") \
.parquet("s3://data/enriched_orders/")
Check spark.sql.shuffle.partitions (default 200). For small datasets,
lower it. For large datasets with skew, raise it.
Choose streaming vs batch
Latency requirement < 1 minute? -> Streaming (Kafka + Flink/Spark Streaming)
Latency requirement 1 hour - 1 day? -> Batch (Airflow + dbt/Spark)
Need both real-time AND historical? -> Lambda (batch + streaming in parallel)
Want one codebase for both? -> Kappa (streaming-only, replay from log)
Streaming is NOT always better. It adds complexity in exactly-once semantics,
state management, late-arriving data, and debugging. Use batch unless you have
a proven real-time requirement.
Common streaming stack: Kafka (ingestion) -> Flink or Spark Structured
Streaming (processing) -> warehouse or serving store (output).
Land raw data in immutable staging, transform into separate layers
Ignoring data skew in Spark
One partition processes 90% of data, job takes hours
Salt keys, broadcast small tables, analyze data distribution first
Skipping schema tests
Bad data silently propagates, discovered by end users
dbt tests, Great Expectations, or custom assertions at every boundary
Over-engineering with streaming
Adds complexity without real-time need
Start with batch, add streaming only for proven sub-minute requirements
Hardcoded dates in queries
Breaks idempotency, prevents backfills
Use Airflow template variables ({{ ds }}) or dbt ref() / source()
No alerting on pipeline failures
Silent failures lead to stale dashboards
Alert on DAG failures, SLA misses, and data freshness breaches
Gotchas
dbt incremental model without unique_key causes duplicates - An incremental model without unique_key set in the config appends new records on every run instead of merging. A re-run after a failure produces duplicate rows that are extremely hard to detect and clean up downstream. Always define unique_key for incremental models.
Airflow catchup=True triggering thousands of backfill runs - If you set catchup=True (the default) on a DAG with a start_date months in the past, Airflow immediately schedules one run per interval from that start date until now. This can flood your workers. Set catchup=False for production DAGs and trigger backfills explicitly via the CLI.
Hardcoded dates break idempotency - SQL queries with WHERE created_at >= '2024-01-01' cannot be safely re-run for different time windows. Use Airflow template variables ({{ ds }}) or dbt source freshness definitions so that re-runs and backfills process the correct partition automatically.
Data skew makes one Spark task run 10x longer - A join key where 80% of rows share one value (e.g., customer_id = NULL or a dominant category) causes one partition to process nearly the entire dataset while others finish immediately. Profile key cardinality with df.groupBy("key").count().orderBy(desc("count")).show(20) before writing join logic.
Streaming over-engineering for batch-compatible requirements - Kafka + Flink adds exactly-once semantics complexity, late-data handling, state backend management, and operational overhead. If the business requirement is "data available within 15 minutes," a scheduled Airflow DAG running every 10 minutes satisfies it with a fraction of the complexity. Start with batch; add streaming only for proven sub-minute latency needs.
References
For detailed patterns and implementation guidance on specific domains, read the
relevant file from the references/ folder:
references/airflow-patterns.md - DAG design patterns, sensors, dynamic DAGs, backfill strategies
references/dbt-patterns.md - model layering, macros, packages, CI/CD for dbt
references/streaming-architecture.md - Kafka, Flink, exactly-once, late data, windowing
Only load a references file if the current task requires it - they are long and
will consume context.
Companion check
On first activation of this skill in a conversation: check which companion skills are installed by running ls ~/.claude/skills/ ~/.agent/skills/ ~/.agents/skills/ .claude/skills/ .agent/skills/ .agents/skills/ 2>/dev/null. Compare the results against the recommended_skills field in this file's frontmatter. For any that are missing, mention them once and offer to install:
