Menu
Workflow and best practices for writing Apache Airflow 3 DAGs. Use when the user wants to write a new DAG, pipeline code, or asks about DAG patterns and conventions.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Debug Airflow DAGs locally using dag.test() in an IDE or command line with pdb. Use when debugging DAG execution, task failures, import errors, or testing DAGs without deploying to Airflow.
Comprehensive guide for testing Apache Airflow DAGs including unit tests, integration tests, load tests, mocking, and best practices for Airflow 3 repositories.
Guide for analyzing pipeline output and debugging Nextflow workflows. Use this when you need to inspect channel contents, trace process execution, or analyze intermediate files.
Research and recommend the best bioinformatics tool for a specific task. Use this when you need to select appropriate tools for bioinformatics analysis based on requirements, performance, integration, and community support.
Guide for interacting with nf-core modules and subworkflows. Use this when you need to discover, view, install, or update nf-core modules/subworkflows in your pipeline.
Guide for creating Mermaid diagrams (.mmd files) from Nextflow pipeline code. Use this when asked to create workflow diagrams from subworkflows or main.nf scripts.
| name | airflow-writing-dags |
| description | Workflow and best practices for writing Apache Airflow 3 DAGs. Use when the user wants to write a new DAG, pipeline code, or asks about DAG patterns and conventions. |
This skill guides you through creating and validating Airflow DAGs using best practices.
The CLI is invoked as airflow. All commands below use this prefix.
+-----------------------------------------+
| 1. DISCOVER |
| Understand codebase & environment |
+-----------------------------------------+
|
+-----------------------------------------+
| 2. PLAN |
| Propose structure, get approval |
+-----------------------------------------+
|
+-----------------------------------------+
| 3. IMPLEMENT |
| Write DAG following patterns |
+-----------------------------------------+
|
+-----------------------------------------+
| 4. VALIDATE |
| Check import errors, warnings |
+-----------------------------------------+
|
+-----------------------------------------+
| 5. TEST (with user consent) |
| Trigger, monitor, check logs |
+-----------------------------------------+
|
+-----------------------------------------+
| 6. ITERATE |
| Fix issues, re-validate |
+-----------------------------------------+
Before writing code, understand the context.
Use file tools to find existing patterns:
Glob for **/dags/**/*.py to find existing DAGsRead similar DAGs to understand conventionsrequirements.txt for available packagesUse airflow CLI commands to understand what's available:
| Command | Description |
|---|---|
airflow dags list | List all DAGs |
airflow dags details <dag_id> | Get DAG details given a DAG id |
airflow dags list-import-errors | List all DAGs that have import errors |
airflow dags list-jobs | List the jobs |
airflow dags list-runs <dag_id> | List DAG runs given a DAG id |
airflow dags next-execution <dag_id> | Get the next logical datetimes of a DAG |
airflow dags pause <dag_id> | Pause DAG(s) |
airflow dags unpause <dag_id> | Resume paused DAG(s) |
airflow dags report | Show DagBag loading report (includes warnings) |
airflow dags reserialize | Reserialize DAGs by parsing the DagBag files |
airflow dags show <dag_id> | Display DAG tasks with their dependencies |
airflow dags show-dependencies | Display DAGs with their dependencies |
airflow dags state <dag_id> <run_id> | Get the status of a dag run |
airflow dags test <dag_id> | Execute one single DagRun locally |
airflow dags trigger <dag_id> | Trigger a new DAG run |
airflow dags delete <dag_id> | Delete all DB records related to the specified DAG |
| Command | Description |
|---|---|
airflow tasks list <dag_id> | List the tasks within a DAG |
airflow tasks state <dag_id> <task_id> <execution_date> | Get the status of a task instance |
airflow tasks states-for-dag-run <dag_id> <run_id> | Get the status of all task instances in a dag run |
airflow tasks test <dag_id> <task_id> | Test a task instance locally |
airflow tasks clear <dag_id> | Clear a set of task instances, as if they never ran |
airflow tasks failed-deps <dag_id> <task_id> | Returns the unmet dependencies for a task instance |
airflow tasks render <dag_id> <task_id> | Render a task instance's template(s) |
| Command | Description |
|---|---|
airflow connections list | List connections |
airflow connections get <conn_id> | Get a connection |
airflow connections add <conn_id> | Add a connection |
airflow connections delete <conn_id> | Delete a connection |
airflow connections test <conn_id> | Test a connection |
airflow connections import <file> | Import connections from a file |
airflow connections export <file> | Export all connections |
airflow connections create-default-connections | Create all default connections from providers |
| Command | Description |
|---|---|
airflow variables list | List variables |
airflow variables get <key> | Get variable |
airflow variables set <key> <value> | Set variable |
airflow variables delete <key> | Delete variable |
airflow variables import <file> | Import variables |
airflow variables export <file> | Export all variables |
| Command | Description |
|---|---|
airflow assets list | List assets |
airflow assets details <asset_uri> | Show asset details |
airflow assets materialize <asset_uri> | Materialize an asset |
| Command | Description |
|---|---|
airflow backfill create <dag_id> | Create a backfill for a dag |
| Command | Description |
|---|---|
airflow providers list | List installed providers |
airflow providers get <provider> | Get detailed information about a provider |
airflow providers hooks | List registered provider hooks |
airflow providers configs | Get information about provider configuration |
airflow providers executors | Get information about executors provided |
airflow providers auth-managers | Get information about auth managers provided |
airflow providers secrets | Get information about secrets backends provided |
airflow providers triggers | List registered provider triggers |
airflow providers links | List extra links registered by the providers |
airflow providers logging | Get information about task logging handlers provided |
airflow providers notifications | Get information about notifications provided |
airflow providers queues | Get information about queues provided |
airflow providers widgets | Get information about registered connection form widgets |
airflow providers behaviours | Get information about registered connection types with custom behaviours |
airflow providers lazy-loaded | Checks that provider configuration is lazy loaded |
| Command | Description |
|---|---|
airflow pools list | List pools |
airflow pools get <pool> | Get pool size |
airflow pools set <pool> <slots> | Configure pool |
airflow pools delete <pool> | Delete pool |
airflow pools import <file> | Import pools |
airflow pools export <file> | Export all pools |
| Command | Description |
|---|---|
airflow config list | List options for the configuration |
airflow config get-value <section> <key> | Print the value of the configuration |
airflow config lint | Lint options for configuration changes while migrating from 2.x to 3.0 |
airflow config update | Update options for configuration changes while migrating from 2.x to 3.0 |
| Command | Description |
|---|---|
airflow db check | Check if the database can be reached |
airflow db check-migrations | Check if migrations have finished |
airflow db migrate | Migrate the metadata database to the latest version |
airflow db clean | Purge old records in metastore tables |
airflow db shell | Run a shell to access the database |
airflow db downgrade | Downgrade the schema of the metadata database |
airflow db reset | Burn down and rebuild the metadata database |
airflow db export-archived | Export archived data from the archive tables |
airflow db drop-archived | Drop archived tables created through the db clean command |
| Command | Description |
|---|---|
airflow version | Show the Airflow version |
airflow info | Show information about current Airflow and environment |
airflow plugins | Dump information about loaded plugins |
airflow cheat-sheet | Display cheat sheet |
airflow jobs check | Check if job(s) are still alive |
Based on discovery, propose:
Get user approval before implementing.
Write the DAG following best practices (see below). Key steps:
Use the airflow CLI as a feedback loop to validate your DAG.
After saving, check for parse errors (Airflow will have already parsed the file):
airflow dags list-import-errors
Common causes: missing imports, syntax errors, missing packages.
airflow dags details <dag_id>
Check: DAG exists, schedule correct, tags set, paused status.
airflow dags report
Look for deprecation warnings or configuration issues.
airflow dags show <dag_id>
Displays tasks with their dependencies. For full metadata combine with airflow dags details <dag_id>.
See the airflow-testing-dags skill for comprehensive testing guidance.
Once validation passes, test the DAG using the workflow in the airflow-testing-dags skill:
airflow dags trigger <dag_id>airflow dags state <dag_id> <run_id> or airflow tasks states-for-dag-run <dag_id> <run_id>airflow tasks states-for-dag-run <dag_id> <run_id> and airflow tasks state <dag_id> <task_id> <execution_date># Ask user first, then:
airflow dags trigger <dag_id>
airflow dags state <dag_id> <run_id>
For the full test -> debug -> fix -> retest loop, see airflow-testing-dags.
If issues found:
airflow dags list-import-errors| Phase | Command | Purpose |
|---|---|---|
| Discover | airflow dags list | List all registered DAGs |
| Discover | airflow connections list | List available connections |
| Discover | airflow variables list | List available variables |
| Discover | airflow providers list | List installed providers |
| Validate | airflow dags list-import-errors | Check for DAG parse/import errors |
| Validate | airflow dags details <dag_id> | Verify DAG metadata and schedule |
| Validate | airflow dags report | Show DagBag report including warnings |
| Validate | airflow dags show <dag_id> | Display task graph and dependencies |
| Test | airflow dags trigger <dag_id> | Manually trigger a DAG run |
| Test | airflow dags state <dag_id> <run_id> | Check status of a DAG run |
| Test | airflow tasks states-for-dag-run <dag_id> <run_id> | Check all task statuses in a run |
| Test | airflow tasks state <dag_id> <task_id> <execution_date> | Check a single task status |
| Debug | airflow tasks failed-deps <dag_id> <task_id> | Show unmet task dependencies |
| Debug | airflow tasks clear <dag_id> | Clear task instances to re-run |
| Debug | airflow dags test <dag_id> | Execute a dry-run DagRun locally |
For code patterns and anti-patterns, see reference/airflow-3-guidance.md and the syntax reference documentation.
Read this reference when writing new DAGs or reviewing existing ones. It covers what patterns are correct (Airflow 3-specific behavior) and what to avoid.