airflow

name	airflow
version	1.0.0
description	Python DAG workflow orchestration using Apache Airflow for data pipelines, ETL processes, and scheduled task automation
author	workspace-hub
category	operations
type	skill
capabilities	["dag_authoring","task_orchestration","scheduling","sensors","operators","hooks","xcoms","variables","connections","kubernetes_deployment","docker_deployment"]
tools	["airflow-cli","docker","kubernetes","helm"]
tags	["airflow","dag","workflow","orchestration","etl","data-pipeline","scheduling","python","automation"]
platforms	["linux","macos","docker","kubernetes"]
related_skills	["yaml-configuration","python-scientific-computing","pandas-data-processing"]
scripts_exempt	true

Airflow

When to Use This Skill

USE when:

• Building complex data pipelines with task dependencies
• Orchestrating ETL/ELT workflows
• Scheduling recurring batch jobs
• Managing workflows with retries and error handling
• Coordinating tasks across multiple systems
• Need visibility into workflow execution history
• Requiring audit trails and lineage tracking
• Building ML pipeline orchestration

DON'T USE when:

• Real-time streaming data (use Kafka, Flink)
• Simple cron jobs (use systemd timers, crontab)
• CI/CD pipelines (use GitHub Actions, Jenkins)
• Low-latency requirements (Airflow has scheduler overhead)
• Simple single-task automation (overkill)
• Need visual workflow design for non-developers (use n8n)

Prerequisites

Installation Options

Option 1: pip (Development)

# Create virtual environment
python -m venv airflow-env
source airflow-env/bin/activate

# Set Airflow home
export AIRFLOW_HOME=~/airflow

# Install Airflow with constraints

*See sub-skills for full details.*
### Development Setup

```bash
# Install development dependencies
pip install apache-airflow[dev,postgres,celery,kubernetes]

# Install testing tools
pip install pytest pytest-airflow

# Install linting
pip install ruff

Version History

Version	Date	Changes
1.0.0	2026-01-17	Initial release with comprehensive workflow patterns

Resources

• Apache Airflow Documentation
• Airflow Best Practices
• Airflow Helm Chart
• Astronomer Guides
• Airflow Providers