data-pipeline-engineering

name	data-pipeline-engineering
description	ETL workflow design, automated data fetching, version tracking, and pipeline orchestration expertise
license	CC-BY-4.0

Data Pipeline Engineering Skill

🔴 AI FIRST Quality Principle

Apply the AI FIRST principle: never accept first-pass quality. Minimum 2 iterations. Read all output, improve every section. No shortcuts.

Purpose

Expert knowledge in designing robust ETL (Extract, Transform, Load) pipelines for automated data processing, focusing on reliability, monitoring, and maintainability.

Core Principles

1. Idempotency - Pipeline runs produce same results
2. Observability - Full visibility into pipeline health
3. Error Recovery - Graceful handling of failures
4. Version Tracking - Track all data changes
5. Monitoring - Real-time pipeline health checks

Enforces

• ETL workflow patterns (Extract → Transform → Load)
• Automated scheduling (cron, GitHub Actions)
• Data versioning and archival
• Pipeline health monitoring
• Error recovery strategies
• Audit logging

When to Use

• Building automated data pipelines
• Scheduling data fetching workflows
• Implementing data versioning
• Monitoring pipeline health
• Designing error recovery

References

• GitHub Actions
• ETL Best Practices

---

Version: 1.0 | Last Updated: 2026-02-06 | Category: Development & Operations