pipeline-migration

name

pipeline-migration

description

Migrate Synapse Data Factory pipeline artifacts to Microsoft Fabric Data Factory. Handles: linked services → Fabric connections, dataset definitions inlined into pipeline activities, global parameters → Variable Libraries, SynapseNotebook activities → TridentNotebook. SSIS, SHIR-only, and Databricks activities are parked. Use when: (1) migrating Synapse pipelines to Fabric Data Factory, (2) converting SynapseNotebook activities to TridentNotebook, (3) translating linked services to Fabric connections, (4) converting global parameters to Fabric Variable Libraries, (5) inlining dataset definitions into Fabric pipeline activities. Triggers: "synapse pipeline to fabric", "data factory pipeline migration", "tridentnotebook pipeline activity", "global parameters to variable library", "linked service to fabric connection", "inline dataset fabric pipeline", "pipeline migration from synapse".

Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry x-ms-fabric-skill: pipeline-migration (az rest: --headers "x-ms-fabric-skill=pipeline-migration"), including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.

CRITICAL NOTES

1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
3. Fabric does NOT have a "Dataset" item type — all dataset properties are inlined into activity typeProperties
4. Linked Services map to Fabric Connections — in pipeline activity JSON, referenceName on the activity's linkedService block uses the Fabric connection display name (not the connection GUID); the connection GUID is used only on Fabric REST API calls
5. Notebook activities change from SynapseNotebook to TridentNotebook and reference notebooks by GUID, not by name
6. Synapse global parameters become a Variable Library item in Fabric, referenced as @pipeline().libraryVariables.<name>. Variable Library Number types are not consumable in pipelines — Synapse Float/Double are mapped to String for runtime compatibility
7. The Validation activity type does not exist in Fabric — it must be rewritten as GetMetadata + IfCondition
8. Triggers are intentionally excluded from this skill — recreate schedules manually in Fabric after migration
9. SSIS package execution, SHIR-exclusive connectors, and Databricks activities are parked — see pipeline-gotchas.md

Synapse Pipelines → Microsoft Fabric Data Factory Migration

Prerequisite Knowledge

These companion documents provide general Fabric REST patterns. Do NOT read them upfront — reference only when a specific phase requires a pattern not already covered in this skill's resource files:

• COMMON-CORE.md — General Fabric REST API patterns, authentication & token audiences, item discovery via JMESPath
• COMMON-CLI.md — az rest / az login CLI patterns, authentication recipes, pipeline run/schedule operations
• ITEM-DEFINITIONS-CORE.md — DataPipeline and VariableLibrary item definition structures (pipeline-content.json, variables.json)
• SPARK-AUTHORING-CORE.md — Fabric notebook item creation (needed when notebook items don't exist yet in Fabric)

For the notebook side of the migration (the notebooks that pipeline activities call), use the companion synapse-migration skill to migrate the notebook content itself.

---

Table of Contents

Topic	Reference
Pre-Migration Assessment (run first)	pipeline-assessment.md
Migration Orchestrator	pipeline-orchestrator.md
API-Driven Migration Workflow	§ API-Driven Migration Workflow
Activity Type Mapping	activity-mapping.md
Notebook Activity Migration (primary focus)	notebook-activity-migration.md
Linked Services → Connections	linked-service-to-connection.md
Dataset Inlining	dataset-inlining.md
Global Parameters → Variable Library	global-parameters-to-variable-library.md
Pipeline Gotchas & Parked Activities	pipeline-gotchas.md
Validation & Testing	validation-testing.md
Migration Report	migration-report.md

Context Loading Guide

IMPORTANT — Load only what you need. Do NOT read all resource files upfront.

When	Read This File
User asks for an assessment, scope, or plan before migrating	pipeline-assessment.md
User asks to migrate a full pipeline workspace	pipeline-orchestrator.md
User asks about activity type mapping or unsupported activities	activity-mapping.md
User has SynapseNotebook activities (most common)	notebook-activity-migration.md
User asks about linked services or connections	linked-service-to-connection.md
User asks about Copy, Lookup, or GetMetadata with datasets	dataset-inlining.md
User has global parameters to convert	global-parameters-to-variable-library.md
User hits SSIS, SHIR, Databricks, or other blockers	pipeline-gotchas.md
Post-migration verification	validation-testing.md
Generating a migration summary	migration-report.md

---

API-Driven Migration Workflow

Authentication

Target	Token Audience
Synapse Data Plane (pipelines, datasets, linked services)	https://dev.azuresynapse.net
Synapse ARM (global parameters, workspace properties)	https://management.azure.com
Fabric REST API (create pipelines, connections, Variable Libraries)	https://api.fabric.microsoft.com

Use az account get-access-token --resource <audience> --query accessToken -o tsv to acquire tokens.

Synapse Data-Plane API Reference

Operation	Endpoint
List all pipelines	GET https://{ws}.dev.azuresynapse.net/pipelines?api-version=2020-12-01
Get pipeline definition	GET https://{ws}.dev.azuresynapse.net/pipelines/{name}?api-version=2020-12-01
List all datasets	GET https://{ws}.dev.azuresynapse.net/datasets?api-version=2020-12-01
Get dataset definition	GET https://{ws}.dev.azuresynapse.net/datasets/{name}?api-version=2020-12-01
List all linked services	GET https://{ws}.dev.azuresynapse.net/linkedservices?api-version=2020-12-01
Get linked service	GET https://{ws}.dev.azuresynapse.net/linkedservices/{name}?api-version=2020-12-01
Get workspace (global parameters)	GET https://management.azure.com/subscriptions/{subId}/resourceGroups/{rg}/providers/Microsoft.Synapse/workspaces/{ws}?api-version=2021-06-01

Fabric API Reference

Operation	Endpoint
List connections	GET https://api.fabric.microsoft.com/v1/connections
Create connection	POST https://api.fabric.microsoft.com/v1/connections
Create pipeline item	POST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items
Update pipeline definition	POST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items/{id}/updateDefinition
Get pipeline definition	POST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items/{id}/getDefinition
Create Variable Library	POST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items (type: VariableLibrary)
List notebooks in workspace	GET https://api.fabric.microsoft.com/v1/workspaces/{wsId}/notebooks

---

Assessment Mode (Optional but Recommended)

Before creating any items in Fabric, run the pipeline assessment to understand scope, complexity, and blockers. The assessment is read-only — it queries Synapse APIs only and produces a markdown report.

Copilot workflow — no Python script file needed:

1. Ask the user: "What is the name of your Synapse workspace?"
2. Auto-discover subscription ID and resource group via az account show and az synapse workspace show
3. Run the assessment code inline from the terminal
4. Print the full report output directly in the chat

When to Use	Action
User wants to understand migration scope before committing	Ask for workspace name → load pipeline-assessment.md → run inline
User asks "what will and won't migrate?"	Run assessment, present the Executive Summary section
User asks for a migration plan or scoping document	Run assessment, print report in chat
User has already decided to migrate	Skip assessment — go straight to Migration Phases below

To also save the report to disk, pass output_path=f"pipeline-assessment-{SYNAPSE_WS}.md" to generate_assessment_report(). The PipelineAssessment objects it produces feed directly into the migration scripts in pipeline-orchestrator.md.

---

Migration Mode (Inline — No Script File Required)

Copilot performs the migration directly from the terminal. No Python files to save or run manually.

Ask the user for:

1. Synapse workspace name (reuse from assessment if already run)
2. Fabric workspace name
3. Which pipelines to migrate — specific names, or * for all
4. Optional name suffix to append to each pipeline in Fabric (e.g. _migrated) — leave blank to keep the original name

Everything else is auto-discovered:

• Subscription ID and resource group via az account show + az synapse workspace show
• Fabric workspace ID via GET /v1/workspaces filtered by display name
• Notebook GUIDs in Fabric via GET /v1/workspaces/{wsId}/notebooks (required for SynapseNotebook → TridentNotebook)
• Connection names in Fabric via GET /v1/connections (when datasets reference linked services)

What is fully automated inline:

• ✅ SynapseNotebook → TridentNotebook — type rename, GUID lookup, remove sparkPool/sessionConfiguration, fix timeout to 12h max
• ✅ All compatible activity types — pass through with minor property adjustments
• ✅ Dataset inlining into activity typeProperties
• ✅ Global parameter expressions rewritten to @pipeline().libraryVariables.<name>
• ✅ Pipeline JSON assembly and deployment to Fabric via REST

Copilot checks before starting — will pause and report if:

• A notebook referenced by a SynapseNotebook activity does not exist in the Fabric workspace yet
• A Fabric Connection is missing for a linked service referenced by dataset activities

Load pipeline-orchestrator.md for the complete inline runner.

---

Migration Phases (Execute in Order)

Phase	Source	Target	Resource
Phase 0	Synapse notebooks (referenced by pipeline activities)	Fabric Notebooks	synapse-migration skill
Phase 1	Synapse global parameters	Fabric Variable Library	global-parameters-to-variable-library.md
Phase 2	Synapse linked services	Fabric Connections	linked-service-to-connection.md
Phase 3	Synapse datasets	Inlined into activities	dataset-inlining.md
Phase 4	Synapse pipeline activities	Fabric pipeline activities	activity-mapping.md + notebook-activity-migration.md
Phase 5	Assembled pipeline JSON	Fabric DataPipeline item	pipeline-orchestrator.md
Final	—	Validation	validation-testing.md

Phase 0 must precede Phase 4: Fabric Notebook GUIDs are needed before TridentNotebook activities can be written. Phase 2 must precede Phase 3: Connection names must exist before they can be referenced in inlined datasets.

---

Activity Type Quick Reference

Full mapping table, before/after examples, and parking decisions are in activity-mapping.md.

Synapse Activity	Fabric Equivalent	Status
SynapseNotebook	TridentNotebook	✅ Migrated — see notebook-activity-migration.md
Copy	Copy	✅ Migrated — datasets inlined
Lookup	Lookup	✅ Migrated — datasets inlined
GetMetadata	GetMetadata	✅ Migrated — datasets inlined
Validation	GetMetadata + IfCondition	✅ Migrated — split into 2 activities
ForEach	ForEach	✅ Compatible
IfCondition	IfCondition	✅ Compatible
Switch	Switch	✅ Compatible
Until	Until	✅ Compatible
Wait	Wait	✅ Compatible
Fail	Fail	✅ Compatible
SetVariable	SetVariable	✅ Compatible
AppendVariable	AppendVariable	✅ Compatible
ExecutePipeline	ExecutePipeline	✅ Compatible — add workspaceId
WebActivity	WebActivity	✅ Compatible
Script	Script

---

Must / Prefer / Avoid

MUST DO

• Migrate notebooks before pipelines — Fabric TridentNotebook activities require notebook GUIDs, not names. Use the synapse-migration skill first
• Create Fabric Connections before building pipeline JSON — linked service names in Synapse become connection references in Fabric; you need the connection names before inlining datasets
• Inline all dataset definitions — Fabric Data Factory has no Dataset item type; all inputs/outputs dataset properties must be embedded in each activity
• Replace @pipeline().globalParameters.<name> with @pipeline().libraryVariables.<name> after creating the Variable Library
• Replace Validation activities with a GetMetadata + IfCondition pair — Validation does not exist as an activity type in Fabric
• Remove sparkPool and sessionConfiguration from migrated notebook activities — pool selection and session config belong in the Fabric Environment attached to the notebook

PREFER

• Variable Library with Value Sets for dev/test/prod environments — use @pipeline().libraryVariables.<name> with environment-specific Value Sets instead of pipeline-level parameters for environment promotion
• Parameterized notebookParameters over hardcoded values in TridentNotebook activities — mirrors Synapse parameterized notebook pattern
• Test notebook activities individually before running the full migrated pipeline — notebook GUIDs are the most common source of failure
• OneLake Lakehouse sources/sinks for Copy activities where applicable — eliminates the need for external connections for data already in OneLake

AVOID

• Do not reference notebooks by name in TridentNotebook activities — use the GUID from the Fabric workspace notebook list
• Do not carry over Synapse triggers — they are not migrated by this skill; recreate schedules in Fabric after validating the pipeline
• Do not attempt to migrate SSIS, Databricks, or AzureBatch activities without reading pipeline-gotchas.md first — these require manual intervention
• Do not hardcode workspace/item GUIDs in pipeline JSON — use Variable Library entries or pipeline parameters so environments can be promoted without editing pipeline JSON
• Do not use @pipeline().globalParameters syntax after migration — this expression path does not exist in Fabric; all migrated global parameters must be accessed via @pipeline().libraryVariables

---

Migration Gotchas — Quick Reference

Full troubleshooting guide is in pipeline-gotchas.md.

#	Flag ID	Issue	Severity	Resolution Summary
PG1	NOTEBOOK_GUID_NOT_FOUND	Notebook not yet migrated to Fabric when building TridentNotebook activity	High	Run synapse-migration skill first; get GUID from GET /v1/workspaces/{wsId}/notebooks
PG2	DATASET_NOT_INLINED	Activity still references a named dataset (not valid in Fabric)	High	Apply dataset-inlining.md patterns to embed dataset properties into activity source/sink
PG3	GLOBAL_PARAM_EXPRESSION	@pipeline().globalParameters.<name> expression left in migrated pipeline	High	Replace with @pipeline().libraryVariables.<name> after creating Variable Library
PG4	VALIDATION_ACTIVITY_UNSUPPORTED	Validation activity type left in pipeline JSON	High	Rewrite as GetMetadata + IfCondition — see activity-mapping.md
PG5	SHIR_CONNECTOR_PARKED	Activity uses a linked service backed by a Self-Hosted Integration Runtime	Medium	Must set up on-premises data gateway in Fabric; see pipeline-gotchas.md
PG6	SSIS_ACTIVITY_PARKED	ExecuteSSISPackage activity cannot be migrated	High	Parked — no Fabric equivalent; see pipeline-gotchas.md for alternatives
PG7	DATABRICKS_ACTIVITY_PARKED	Databricks activity type has no Fabric native equivalent	High	Parked — use Databricks REST API via WebActivity as workaround; see pipeline-gotchas.md
PG8	SPARKPOOL_REF_ORPHANED	sparkPool / targetBigDataPool reference left in TridentNotebook activity	Medium	Remove sparkPool and sessionConfiguration blocks; pool config belongs in Fabric Environment
PG9	EXECUTE_PIPELINE_NO_WORKSPACE

---

Post-Migration: What's Next

After pipeline migration, hand off to these companion skills and tools:

Task	Skill / Tool
Migrate notebook content (mssparkutils → notebookutils, linked services)	synapse-migration skill
Schedule migrated pipelines	COMMON-CLI.md § Job Scheduling
Monitor pipeline runs	Fabric workspace → Monitor hub
Build new Fabric pipelines	Refer to ITEM-DEFINITIONS-CORE.md § DataPipeline
Explore migrated Lakehouse data post-pipeline run	spark-cli or sqldw-cli skill

---

Examples

SynapseNotebook → TridentNotebook activity (before/after)

{
  "name": "Run_Notebook",
  "type": "SynapseNotebook",
  "typeProperties": {
    "notebook": {"referenceName": "MyNotebook", "type": "NotebookReference"},
    "sparkPool": {"referenceName": "BigPool", "type": "BigDataPoolReference"}
  }
}

After migration (GUID from GET /v1/workspaces/{wsId}/notebooks):

{
  "name": "Run_Notebook",
  "type": "TridentNotebook",
  "typeProperties": {
    "notebookId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "workspaceId": "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"
  }
}

Global parameter expression rewrite

Before (Synapse):  @pipeline().globalParameters.batchDate
After (Fabric):    @pipeline().libraryVariables.batchDate

Dataset inlining: Copy activity — dataset reference → connection display name

Synapse dataset InputBlobDataset (will be inlined and removed):

{
  "type": "AzureBlob",
  "linkedServiceName": {"referenceName": "AzureBlobLinkedService"},
  "typeProperties": {"folderPath": "input/", "fileName": "data.csv"}
}

Synapse Copy activity (before — references dataset by name):

{
  "name": "CopyData", "type": "Copy",
  "inputs": [{"referenceName": "InputBlobDataset", "type": "DatasetReference"}],
  "typeProperties": {"source": {"type": "BlobSource"}, "sink": {"type": "BlobSink"}}
}

After migration (connection display name for AzureBlobLinkedService is My ADLS Connection):

{
  "name": "CopyData", "type": "Copy",
  "typeProperties": {
    "source": {"type": "BlobSource", "storeSettings": {"type": "AzureBlobStorageReadSettings"}},
    "sink":   {"type": "BlobSink",   "storeSettings": {"type": "AzureBlobStorageWriteSettings"}}
  },
  "linkedService": {"referenceName": "My ADLS Connection", "type": "LinkedServiceReference"}
}

See activity-mapping.md and notebook-activity-migration.md for full before/after examples.