data_engineer_claude_skills

Umbrella skill for running a data project end-to-end through its lifecycle stages — discover sources → profile the data → architect the platform → build the medallion pipeline → refactor the code. Use this whenever the user is kicking off a new data project, asks "where do I start" or "what are the steps", or is somewhere mid-lifecycle and unsure which stage skill applies. This skill ROUTES to the stage sub-skills (data-sourcing, data-profiling, data-architecture, medallion-design, notebook-refactor) and sequences them, pulling in the right one for the user's current stage.

Data collection and readiness — the step BEFORE a pipeline exists. Catalog candidate data sources for an objective, rank each by importance/impact, and record where to get it (internal system vs. open/public dataset), how to access it, its refresh cadence, licensing, and readiness blockers. Use this skill whenever a project is starting and the sources aren't decided yet, when the user asks "what data do I need for X and where do I get it", when weighing internal vs. open-source data, or when assessing whether a source is accessible/usable before committing to a build. This sits upstream of pipeline-design — it decides WHAT to ingest before deciding HOW.

Profile and map raw data BEFORE designing a schema. One-time exploratory analysis to learn the true shape of a dataset — row/column counts, null rates, cardinality, value distributions, ranges, data types, candidate keys, duplicates, referential relationships — then a source-to-target field mapping and an ER diagram. Use this skill whenever the user has data in hand and needs to understand it before modeling, is about to design a schema, asks "what does this data actually look like", needs to find the primary/composite key of an unfamiliar table, or must map source fields to a target model. This is exploratory and one-time; ongoing production validation is data-quality's job, and modeling patterns are schema-design's job.

Umbrella skill for shaping data once it has landed — warehouse schema design (star/snowflake/OBT/SCD/grain), SQL for analytics (window functions, CTEs, optimization), dbt model layers/tests/macros/incrementals, and Python transforms (pandas/Polars/PySpark performance). Use this whenever the user is designing tables, writing or reviewing analytical SQL, building dbt models, or writing Python data transformations and it isn't yet clear which tool dominates. This skill ROUTES to the focused sub-skills (schema-design, sql-patterns, dbt-patterns, python-data-patterns) and pulls in more than one when a task spans them. Trigger on: fact/dimension design, table grain, SCD, window functions, deduplication, dbt refs/materializations, or slow/memory-heavy DataFrame code.

dbt model design, ref chains, sources, tests, macros, incremental strategies, materializations, and documentation best practices. Use this skill whenever the user is writing or reviewing dbt models, configuring dbt tests, designing model layers (staging/intermediate/marts), asking about incremental models, choosing materializations, writing macros, setting up sources.yml, or troubleshooting dbt run/test failures. Also trigger when the user mentions dbt refs, lineage graphs, model dependencies, dbt Cloud, the dbt CLI, or when they want to transform data already in the warehouse using SQL. If the project uses dbt at all, this skill should be active for any transformation questions.

Pandas, Polars, and PySpark idioms for production data engineering — chunked reads, memory-safe transforms, vectorized operations, type optimization, and performance patterns. Use this skill whenever the user is writing a Python data transformation script and running into memory issues, slow performance, or correctness bugs with large datasets. Also trigger when the user asks how to handle large CSV/Parquet files, process data in batches, use Polars instead of Pandas, optimize a PySpark job, or reduce DataFrame memory usage. If you see someone iterating row-by-row over a DataFrame, this skill should trigger immediately.

Data modeling for analytical workloads — star schema, snowflake schema, one big table (OBT), slowly changing dimensions (SCD), normalization tradeoffs, grain definition, and surrogate key strategies. Use this skill whenever the user is designing or reviewing a data warehouse schema, planning a fact/dimension table layout, deciding how to model a business entity (customer, order, event, product), or asking how to handle historical changes to dimension attributes. Also trigger when the user asks about dbt model design, table granularity, or how to structure data for BI tools like Looker, Tableau, or Power BI. Get this right before building the pipeline — a bad schema is expensive to fix later.

Best-practice SQL for analytical workloads — window functions, CTEs, query optimization, partitioning strategies, and anti-patterns to avoid. Use this skill whenever the user is writing or reviewing a SQL query that goes beyond a basic SELECT, especially on BigQuery, Snowflake, Redshift, or DuckDB. Trigger on mentions of aggregations, ranking, running totals, session analysis, lag/lead comparisons, deduplication, slowly-changing lookups, or any time the user asks "how do I write a query for X". Also trigger when a query looks slow, returns wrong results, or the user asks for a code review of existing SQL.

Umbrella skill for where data pipelines run and what they cost — AWS/GCP/Azure infrastructure (S3/GCS/ADLS layout, BigQuery/Redshift/Snowflake selection, IAM, managed-service choice, performance tuning) and cost control (bytes scanned, partition pruning, slot/credit management, storage tiering, budgets and alerts). Use this whenever the topic is cloud data infrastructure or the cloud data bill. This skill ROUTES to the focused sub-skills (cloud-infra-data, cost-optimization-data) and pulls in both when a task spans architecture and cost. Trigger on: deploying a pipeline to cloud, choosing between managed warehouses, bucket/partition layout, IAM for data access, "our BigQuery bill jumped", expensive queries, or slot/credit utilization.

AWS/GCP/Azure data infrastructure — S3/GCS/ADLS partitioning, BigQuery slot management, Redshift spectrum, Snowflake warehouses, IAM roles for data access, cost optimization, and managed service selection. Use this skill whenever the user is deploying a pipeline to cloud, choosing between managed data services, configuring storage for a data lake, setting up IAM/permissions for pipelines, asking about BigQuery pricing, Redshift vs. BigQuery vs. Snowflake, S3 bucket layout, or cloud-specific performance tuning. Also trigger when the user mentions cloud costs, slow BigQuery queries, Redshift concurrency scaling, storage formats in the cloud, or cross-account data access. If it touches cloud + data together, this skill should be active.

Query cost analysis, partition pruning, slot reservation strategies, storage tiering, and cloud data warehouse cost reduction. Use this skill whenever the cloud data bill is unexpectedly high, a specific query is scanning too much data, the team wants to understand what's driving BigQuery/Snowflake/Redshift costs, or when choosing between on-demand vs. reserved capacity. Also trigger when the user mentions bytes scanned, slot utilization, query cost, storage costs, Redshift concurrency, Snowflake credits, or when trying to set up cost alerts and budgets. If someone says "our BigQuery bill jumped" or "this query is expensive", this skill should be active immediately.

Objective-first data architecture design and decision framework. Establish the target object FIRST (BI, ML, or both), elicit requirements and constraints (cloud vs. on-prem, resource/compute budget, team skills, latency and freshness SLAs, compliance), then design the layered architecture and recommend a model — evaluating cost-effectiveness per layer across Azure, AWS, and GCP. Use this skill whenever the user is planning a new data platform or a major redesign, deciding cloud vs. on-prem, choosing between clouds/warehouses at the architecture level, or asking "how should I structure this whole thing". This is a ROUTER/FRAMEWORK: it owns objective and constraint elicitation and the layer design, and delegates cloud-service specifics to cloud-infra-data and pricing detail to cost-optimization-data.

Moving data between systems safely — cutover planning, backfill strategies, dual-write patterns, validation, rollback procedures, and zero-downtime migration techniques. Use this skill whenever the team is migrating from one database or warehouse to another (MySQL → Snowflake, Redshift → BigQuery, on-prem → cloud), replacing a legacy pipeline, doing a major schema change on a live table, or planning a cutover that cannot have downtime. Also trigger when the user asks about dual-write, shadow reads, data validation across systems, incremental vs. full migration, or how to safely retire an old system. If the phrase "migrate", "move data", "cutover", "legacy system", or "replace the old pipeline" appears, this skill should be active.

Umbrella skill for moving data from source to destination — end-to-end ETL/ELT design, DAG orchestration, real-time streaming, and system-to-system migration. Use this whenever the user is building, scheduling, debugging, or migrating a pipeline and it isn't yet clear which sub-area dominates. This skill ROUTES to the focused sub-skills (pipeline-design, orchestration-patterns, streaming-patterns, data-migration) and pulls in more than one when a task spans them (e.g. "design a streaming pipeline and orchestrate its backfill"). Trigger on: new ingestion job, batch vs. streaming choice, Airflow/Prefect/Dagster DAGs, CDC, backfill, cutover, dual-write, or "move data from X to Y".

Design a medallion (bronze/silver/gold) ETL architecture interactively, objective-first. List the available data, confirm the objective, then design GOLD first to match the objective and get the user to review it before moving down to silver, then bronze (top-down default) — or bronze-up if the user asks. Asks the user to confirm at each layer boundary rather than designing all three in one shot. Wraps the reusable utils/ library (bronze.py, silver.py, scd.py, watermark.py, metadata.py, quality.py). Use this skill whenever the user wants a bronze/silver/gold or medallion/lakehouse layout, is building a layered Delta/warehouse pipeline, or wants an objective-driven top-down layer design. For extraction/idempotency mechanics reference pipeline-design; this owns the layered design conversation.

Airflow/Prefect/Dagster DAG design — task dependencies, retries, SLAs, backfill strategies, sensors, and failure recovery. Use this skill whenever the user is building or debugging a scheduled pipeline with multiple steps, asking how to handle task failures, setting up retries or alerts, designing a DAG structure, choosing between orchestrators, or dealing with backfill/reprocessing of historical data. Also trigger when the user mentions Airflow operators, Prefect flows, Dagster assets, task queues, or pipeline scheduling — even if they don't say "orchestration" explicitly. If a pipeline has more than two steps and needs to run on a schedule, this skill should be active.

Design ETL/ELT pipelines end-to-end — source connectors, extraction strategies, transform logic, load patterns, idempotency, scheduling, and error handling. Use this skill whenever the user is starting a new ingestion job, planning how data moves from a source (REST API, database, file, webhook, message queue) into a data warehouse or data lake. Also trigger when the user asks about pipeline architecture, incremental vs. full loads, backfill strategies, CDC, retry logic, or orchestration choices (Airflow, Prefect, dbt). This skill should feel like pairing with a senior data engineer on day one of a new pipeline project.

Kafka, Flink, Kinesis, and Spark Structured Streaming design — consumer groups, partitioning, exactly-once semantics, lag monitoring, windowing, and late-arriving data. Use this skill whenever the user needs real-time or near-real-time data processing, is redesigning a batch pipeline into streaming, asks about event-driven architectures, or mentions Kafka topics, consumer lag, checkpointing, watermarks, or stream-table joins. Also trigger when the user says batch is "too slow", stakeholders want "live" dashboards, or the pipeline needs to react to events as they happen rather than on a schedule. If latency requirements are under a few minutes, this skill should be active.

Define and enforce schema contracts between producer and consumer teams — field types, nullability, allowed values, versioning, breaking vs. non-breaking changes, and change detection patterns. Use this skill whenever two teams or services share a dataset and upstream changes keep breaking the downstream silently, when a team wants to formalize what a dataset "promises" to its consumers, or when setting up schema validation at pipeline boundaries. Also trigger when the user asks about schema evolution, backward/forward compatibility, schema registries, Great Expectations for inter-team contracts, or when a producer is about to make a "small" schema change and you want to assess its downstream impact. If upstream and downstream are owned by different people, this skill should be active.

Data lineage tracking, PII tagging, access control policies, data catalog metadata standards, retention policies, and audit logging for regulatory compliance. Use this skill whenever the company is subject to PDPA, GDPR, HIPAA, or any data privacy regulation, when an audit requires proof of who accesses what data, when PII fields need to be identified and classified in a dataset, when setting up column-level access control, or when building a data catalog. Also trigger when someone asks about data masking, anonymization, right-to-erasure workflows, role-based data access, or data lineage from source to BI tool. If the word "compliance", "audit", "PII", "sensitive data", or "regulation" appears, this skill should be active.

Write systematic data quality checks — validation rules, Great Expectations suites, dbt tests, anomaly detection, null/type/range/referential integrity assertions, and monitoring patterns for production pipelines. Use this skill whenever the user is dealing with bad data in a pipeline, setting up validation before or after a load step, adding tests to dbt models, writing Great Expectations expectations, or trying to detect when upstream data has changed shape. Also trigger when stakeholders keep finding incorrect numbers, when a pipeline silently loads garbage, or when the user asks "how do I make sure my data is correct". Prevention is cheaper than debugging.

Umbrella skill for making data correct, trustworthy, and compliant — validation and quality checks (Great Expectations, dbt tests, anomaly/null/range/referential assertions), producer↔consumer schema contracts (versioning, breaking-change detection), and governance (PII tagging, lineage, access control, retention, audit for PDPA/GDPR/HIPAA). Use this whenever the concern is "is this data right, safe, and allowed" rather than moving or shaping it. This skill ROUTES to the focused sub-skills (data-quality, data-contracts, data-governance) and pulls in more than one when a task spans them. Trigger on: bad data in a pipeline, silent upstream schema changes breaking downstream, PII/compliance/audit, or "how do I make sure my data is correct".

Feature store patterns, training/serving skew prevention, feature pipelines for ML teams, point-in-time correct joins, and bridging data engineering with MLOps conventions. Use this skill whenever an ML team needs feature pipelines, when building a feature store or deciding whether to use one, when there's a training/serving skew problem (model performance in production differs from validation), when features need to be shared across multiple models, or when designing point-in-time correct feature computation. Also trigger when the user mentions feature stores (Feast, Tecton, Hopsworks), label leakage, backfilling features, offline/online store separation, or when data engineering work feeds directly into model training. If the words "features", "training set", "model pipeline", or "MLOps" appear alongside data engineering questions, this skill should be active.

Refactor a Jupyter/.ipynb notebook from top-to-bottom exploratory sprawl into small, named, reviewable functions with clear inputs and outputs. Extract subfunctions, separate config/params from logic, remove hidden cell-order dependencies and global-state leakage, add docstrings and light tests, and make the notebook re-runnable top-to-bottom and diff-friendly. Use this skill whenever the user has a messy notebook they want cleaned up, wants notebook code turned into functions or a module, is preparing a notebook for review/handoff/production, or asks to make an .ipynb readable or testable. This is about code STRUCTURE and reviewability; performance and correctness of data code is python-data-patterns' job.

Translate pipeline metrics, SLA breaches, data quality failures, and incidents into clear non-technical summaries for business stakeholders. Use this skill whenever data was late, wrong, or missing and someone needs to communicate what happened to a manager, director, or business team. Also trigger when writing incident reports, SLA breach notifications, data quality summaries, pipeline health updates, or any communication where the audience doesn't know what Airflow or BigQuery is. If the user needs to explain a technical data failure to a non-technical person, this skill should be active.