merge-duplicate-pipelines

name

merge-duplicate-pipelines

description

Finds Orchestra pipelines that are really the same process duplicated — across environments (`_prod`/`_staging`/`_dev`/`-uat` naming, or Orchestra's native Environments) or conceptually (same task graph copy-pasted per customer/region/business-unit under unrelated names) — then drafts one consolidated pipeline using Environment overlays, `${{ ENV.* }}`, inputs, conditionals, or a MetaEngine matrix instead of the duplication. Use when the user wants to "consolidate", "merge", "dedupe", or "unify" pipelines, asks "why do I have three copies of this pipeline", or wants to act on an account-health-check finding about one-pipeline-per-environment, duplicated tasks, or hardcoded environment values. Also trigger when handed an existing account-review report flagging those. Always shows evidence and a drafted YAML before touching anything, and asks per duplicate set whether to report, create/PR the unified pipeline, or also pause the originals — never merges, deletes, or pauses without that go-ahead.

Merge Duplicate Pipelines

Find pipelines in an Orchestra workspace that are the same process duplicated — across environments or conceptually — and consolidate each duplicate set into one pipeline with the right parameterisation in place of the copy-paste. Detection is read-only; anything that changes a pipeline requires an explicit choice from the user for that specific set.

References

• ../../references/orchestra/mcp/tools-quick-ref.md — MCP tool names and arguments.
• ../../references/orchestra/pipeline/yaml-authoring.md — schema and variable syntax (${{ ENV.* }}, ${{ inputs.* }}, ${{ MATRIX.* }}, conditions).
• references/detection-and-patterns.md — read this before drafting anything. The similarity heuristic in full, the ENV-LABELED vs CONCEPTUAL classification, and before/after YAML for each consolidation pattern.
• ../account-health-check/references/best-practices-checklist.md checks 1.1 (one pipeline per process, not per environment), 1.3 (MetaEngine instead of duplicated tasks/pipelines), 1.9 (inputs/env vars/conditionals over duplication), and 2.1 (don't run one pipeline per environment) — this skill is the fix for those findings.

Workflow

Step 0 — Fresh detection, or acting on an existing report?

If the user hands you an existing account-review report, or pastes findings that name checks 1.1/1.3/1.9/2.1, read it for the pipelines it already flagged and skip straight to Step 3 with those as your starting candidate set — no need to re-scan the account. Otherwise start at Step 1.

Step 1 — Access and scope

Same access pattern as account-health-check: use the Orchestra MCP if connected (list_pipelines, get_pipeline); otherwise fall back to the REST API with $ORCHESTRA_API_KEY (https://app.getorchestra.io/api/engine/public). If neither is available, point the user at the MCP setup docs and stop.

Tell the user which workspace you're scanning and confirm scope: full account, or a named subset. For large workspaces, default to the live + Git-backed set (every unpaused, recently-run pipeline plus every Git-backed one) for the same reason account-health-check does — reading every definition has a real cost — and say what you skipped.

Step 2 — Gather pipeline data

1. list_pipelines for the inventory: name, alias, storageProvider, paused, numTasks, schedule. Parse schedule/sensors/webhook out of their JSON-encoded string form before using them (see the trap noted in account-health-check).
2. get_pipeline (or the REST fallback) for the full definition of every in-scope pipeline. This is what the similarity signatures in Step 3 are built from — there's no cheaper source, so this is the expensive step; that's why scope matters.
3. If connected, list_environments/get_environment to see whether the account already has Orchestra Environments configured. This changes the framing of a finding: "you haven't set up Environments yet" vs. "you have Environments, but this pipeline isn't using them."

Step 3 — Detect candidate duplicate sets

Run both passes from references/detection-and-patterns.md and union the results:

• Naming pass — strip environment-like tokens from name/alias and group exact/near matches. Cheap, high-precision, catches the obvious case.
• Structural pass — for pipelines the naming pass didn't group, compare task-graph signatures (integration + job, ignoring IDs/names/param values) with a similarity score. This pass is explicitly tolerant of partial differences — a pipeline that's a subset or superset of another (an extra QA step in staging, an extra notify step in prod) still counts as a strong candidate; note the delta rather than requiring an exact match. See the thresholds in the reference file — don't force a merge on a weak/ambiguous match, surface it and ask instead.

For every candidate set, classify it ENV-LABELED or CONCEPTUAL per the reference file — this determines which pattern you draft next.

Step 4 — Draft the consolidated pipeline per set

Per the classification:

• ENV-LABELED → Pattern A (Environment overlay). Replace hardcoded env-specific values with ${{ ENV.VAR }} using the same variable name across every variant. Tasks present in only some variants become their own task group gated with condition (Pattern B). Name the environments in the draft (schedule environment: pins, any ENVIRONMENT_NAME reference) using the real names/IDs from Step 2's list_environments call whenever that succeeded — don't guess a name like prod/staging when the real one was available. Only fall back to a naming-pattern guess when Environments couldn't be reached, and flag it explicitly as unconfirmed.
• CONCEPTUAL → Pattern C (MetaEngine matrix) if the differing dimension is enumerable (customers, regions); otherwise pipeline inputs + a Pattern-B conditional group for the handful of variant-specific steps.
• Carry forward every task that exists in any member of the set. Don't silently drop a step that only some variants have — gate it, and call out the gating explicitly in the report so the user can confirm the condition matches real intent rather than a guess.
• Validate the draft: orchestra-cli validate <path> if available, else MCP validate_pipeline.

Step 5 — Present and ask before doing anything

Detection and drafting are read-only up to this point. For each candidate set, show:

1. The member pipelines and the similarity evidence (score, which tasks matched, which differed and where they ended up in the draft).
2. The drafted unified YAML (or a diff against the largest member).
3. Any manual prerequisites (environment variables to register, secrets to re-enter in the UI, Environments that don't exist yet).

Then ask the user which action to take for that specific set — there is no default:

1. Report only — write the draft to a file, make no pipeline changes.
2. Create/PR the unified pipeline, leave originals running — validate on a branch (or a draft Orchestra-backed pipeline), but don't touch the duplicates.
3. Do the above, and also pause the originals once the new pipeline validates — pause only, never delete.
4. Skip this set.
5. Also set up the missing Environments/variables — only offer this when Step 2 found a real gap (an Environment that doesn't exist, or a variable this draft needs that isn't registered anywhere yet). Creating a brand-new Environment is comparatively safe; adding a variable to one that already exists is not (it's a full-value-set replace, not a merge) — see references/detection-and-patterns.md §7 for exactly when each is safe to do versus when to leave it as a manual UI step instead.

Every set gets its own answer; don't assume the same choice applies to all of them. Option 5 is additive to whichever of 1–4 was chosen, not a replacement for it.

Step 6 — Execute the chosen action

The unified pipeline is always a new, additional pipeline — never one of the original duplicates rewritten in place. That keeps "create the consolidation" and "retire the originals" as two separable, individually-approved actions rather than one irreversible step:

• Git-backed: new branch, new YAML file, open a PR, validate with a branch run (start_pipeline with branch=...) — same gate as triage-orchestra-pipeline. Nothing merges to main until the user approves the PR, and the original duplicate files are untouched by this PR.
• Orchestra-backed (storageProvider: ORCHESTRA): create_pipeline for the new unified pipeline. Do not call update_pipeline/migrate_pipeline on any of the original duplicate pipelines here — those tools are only used in the pausing step below, and only to flip paused, never to rewrite a definition. (Aside: if create_pipeline or a later update_pipeline call 422s on storage_provider with extra_forbidden, that's a known Orchestra MCP quirk — retry the equivalent call via migrate_pipeline instead, which omits that field.)
• Pausing originals (only if the user chose option 3, and only after the new pipeline has validated): pause via update_pipeline/migrate_pipeline — a paused: true toggle on the existing duplicate, nothing else in its definition changes. Never delete a pipeline — that's not offered as an option for a reason.

Step 7 — Report

Write (or append to) orchestra-pipeline-consolidation.md: one section per candidate set with the evidence, classification, action taken or deferred, links to the PR/pipeline, and any outstanding manual steps. Chat summary: how many sets found, what was done per set, what's still pending on the user.

Notes

• Never delete a pipeline, under any option.
• Cite the specific checklist item in the report: 1.1/2.1 for environment-suffix duplication, 1.3 for enumerable duplication that should be a matrix, 1.9 for hardcoded values that should be ${{ ENV.* }}/${{ inputs.* }}.
• A weak or ambiguous structural match usually means two pipelines that look similar but do genuinely different things. Don't force a merge to hit a number — surface the comparison, say what's uncertain, and let the user decide whether it's really a duplicate.