senior-data-engineer

name	senior-data-engineer
title	Senior Data Engineer Skill Package
description	World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.
domain	engineering
subdomain	data-engineering
difficulty	advanced
time-saved	TODO: Quantify time savings
frequency	TODO: Estimate usage frequency
use-cases	["Designing data pipelines for ETL/ELT processes","Building data warehouses and data lakes","Implementing data quality and governance frameworks","Creating analytics dashboards and reporting","Building real-time streaming pipelines with Kafka and Flink","Implementing exactly-once streaming semantics","Monitoring streaming quality (consumer lag, data freshness, schema drift)"]
related-agents	[]
related-skills	[]
related-commands	[]
orchestrated-by	[]
dependencies	{"scripts":[],"references":[],"assets":[]}
compatibility	{"python-version":"3.8+","platforms":["macos","linux","windows"]}
tech-stack	["Python","SQL","Apache Spark","Airflow","dbt","Apache Kafka","Apache Flink","AWS Kinesis","Spark Structured Streaming","Kafka Streams","PostgreSQL","BigQuery","Snowflake","Docker","Schema Registry"]
examples	[{"title":"Example Usage","input":"TODO: Add example input for senior-data-engineer","output":"TODO: Add expected output"}]
stats	{"downloads":0,"stars":0,"rating":0,"reviews":0}
version	v2.0.0
author	Claude Skills Team
contributors	[]
created	"2025-10-20T00:00:00.000Z"
updated	"2025-12-16T00:00:00.000Z"
license	MIT
tags	["architecture","data","design","engineer","engineering","senior","streaming","kafka","flink","real-time"]
featured	false
verified	true

Senior Data Engineer

Core Capabilities

• Batch Pipeline Orchestration - Design and implement production-ready ETL/ELT pipelines with Airflow, intelligent dependency resolution, retry logic, and comprehensive monitoring
• Real-Time Streaming - Build event-driven streaming pipelines with Kafka, Flink, Kinesis, and Spark Streaming with exactly-once semantics and sub-second latency
• Data Quality Management - Comprehensive batch and streaming data quality validation covering completeness, accuracy, consistency, timeliness, and validity
• Streaming Quality Monitoring - Track consumer lag, data freshness, schema drift, throughput, and dead letter queue rates for streaming pipelines
• Performance Optimization - Analyze and optimize pipeline performance with query optimization, Spark tuning, and cost analysis recommendations

Key Workflows

Workflow 1: Build ETL Pipeline

Time: 2-4 hours

Steps:

1. Design pipeline architecture using Lambda, Kappa, or Medallion pattern
2. Configure YAML pipeline definition with sources, transformations, targets
3. Generate Airflow DAG with pipeline_orchestrator.py
4. Define data quality validation rules
5. Deploy and configure monitoring/alerting

Expected Output: Production-ready ETL pipeline with 99%+ success rate, automated quality checks, and comprehensive monitoring

Workflow 2: Build Real-Time Streaming Pipeline

Time: 3-5 days

Steps:

1. Select streaming architecture (Kappa vs Lambda) based on requirements
2. Configure streaming pipeline YAML (sources, processing, sinks, quality)
3. Generate Kafka configurations with kafka_config_generator.py
4. Generate Flink/Spark job scaffolding with stream_processor.py
5. Deploy and monitor with streaming_quality_validator.py

Expected Output: Streaming pipeline processing 10K+ events/sec with P99 latency < 1s, exactly-once delivery, and real-time quality monitoring

World-class data engineering for production-grade data systems, scalable pipelines, and enterprise data platforms.

Overview

This skill provides comprehensive expertise in data engineering fundamentals through advanced production patterns. From designing medallion architectures to implementing real-time streaming pipelines, it covers the full spectrum of modern data engineering including ETL/ELT design, data quality frameworks, pipeline orchestration, and DataOps practices.