create-orchestra-pipeline

name

create-orchestra-pipeline

description

Create, validate, and remediate Orchestra pipeline YAML files. Use when asked to build a new pipeline, add tasks to an existing pipeline, fix pipeline validation errors, or author Orchestra workflow definitions from a description. Trigger on phrases like "create a pipeline", "add a dbt task", "write orchestra yaml", "fix validate errors", or when editing files under orchestra/ or similar pipeline directories.

Create Orchestra Pipeline

Author or update an Orchestra version: v1 pipeline YAML, validate it, and fix validation errors.

References

• ../../references/orchestra/pipeline/yaml-authoring.md — schema, integrations, variables, optional sections
• ../../references/orchestra/pipeline/examples.md — multi-stage patterns (warehouse → LLM → messaging, agents)
• ../../references/orchestra/mcp/tools-quick-ref.md — validate_pipeline, create_pipeline, update_pipeline
• Orchestra docs for integration parameters not covered in the reference

Workflow

Step 1 — Understand the request

From the user message, determine:

• Purpose, integrations, and data flow
• Target file path (default: orchestra/<descriptive-name>.yml in the current repo)
• Connections, schedules, inputs, alerts, or matrix requirements

If no filename is given, derive a short kebab-case name from the pipeline purpose.

When editing an existing pipeline, check whether it's Git-backed or Orchestra-backed before deciding how to apply the change — list_pipelines metadata includes storageProvider. For Git-backed pipelines, edit the repo YAML directly (the repo is the source of truth); update_pipeline cannot write to them. For Orchestra-backed pipelines with no local YAML, use the MCP tools instead of authoring a file.

Step 2 — Match repo conventions

List existing pipeline YAML (typically orchestra/, or paths the user names). Read one or two pipelines that use similar integrations before writing — match task group and task ID style, connection references, and schedule format.

Step 3 — Write or edit the YAML

Follow ../../references/orchestra/pipeline/yaml-authoring.md for structure, required fields, integration table, and variable syntax. Omit empty tags arrays.

Editorial defaults, unless the user asks otherwise:

• Prefer cron or webhook triggers over sensors where the source integration supports it.
• Don't hardcode a connection; omit it and let Orchestra pick the default, or reference an existing environment variable (${{ ENV.VAR }}) when one already covers that connection.
• Add a failure alert as a matter of course — most pipelines should have one.
• For matrices, define the inputs list once on the task group, then reference values in task parameters as ${{ MATRIX.key }} rather than repeating the list per task.

Step 4 — Validate

Run local validation when orchestra-cli is available:

orchestra-cli validate <path/to/pipeline.yml>

If only Orchestra MCP is connected, use validate_pipeline with the YAML body instead.

Step 5 — Remediate errors

For each validation error, apply the fixes in the table in yaml-authoring.md. Re-validate until clean.

Step 6 — Report

Summarise in a short paragraph or bullet list:

1. File path created or modified
2. Stages and tasks
3. Connections or environment variables to configure in the Orchestra UI
4. Placeholder values the user must replace

Keep the summary concise.

Notes

• Pipelines can represent both data workflows and AI agent workflows; use PYTHON / PYTHON_EXECUTE_SCRIPT with build_command and project_dir for in-repo agent entrypoints.
• After saving YAML in repos that use Cursor hooks, orchestra-cli validate may run automatically on *.yml / *.yaml — still run validation explicitly when unsure.
• For Git-backed pipelines, committing YAML does not deploy until the repo is connected in Orchestra; call out any UI setup the user still needs.
• Task group and task IDs only need to be unique strings, not real UUIDs.
• Don't add run_inputs to a trigger unless asked — that's for setting inputs dynamically per trigger, not a default every pipeline needs.