dataflows-authoring-cli

> **Update Check — ONCE PER SESSION (mandatory)** > The first time this skill is used in a session, run the **check-updates** skill before proceeding. > - **GitHub Copilot CLI / VS Code**: invoke the `check-updates` skill. > - **Claude Code / Cowork / Cursor / Windsurf / Codex**: compare local vs remote package.json version. > - Skip if the check was already performed earlier in this session. > **CRITICAL NOTES** > 1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering > 2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering # Dataflows Gen2 — Authoring via CLI ## Table of Contents | Task | Reference | Notes | |---|---|---| | Finding Workspaces and Items in Fabric | [COMMON-CLI.md § Finding Workspaces and Items in Fabric](../../common/COMMON-CLI.md#finding-workspaces-and-items-in-fabric) | **Mandatory** — *READ link first* [needed for finding workspace id by its name or item id by its name, item type, and workspace id] | | Fabric Topology & Key Concepts | [COMMON-CORE.md § Fabric Topology & Key Concepts](../../common/COMMON-CORE.md#fabric-topology--key-concepts) || | Environment URLs | [COMMON-CORE.md § Environment URLs](../../common/COMMON-CORE.md#environment-urls) || | Authentication & Token Acquisition | [COMMON-CORE.md § Authentication & Token Acquisition](../../common/COMMON-CORE.md#authentication--token-acquisition) | Wrong audience = 401; read before any auth issue | | Core Control-Plane REST APIs | [COMMON-CORE.md § Core Control-Plane REST APIs](../../common/COMMON-CORE.md#core-control-plane-rest-apis) | Includes pagination, LRO polling, and rate-limiting patterns | | Definition Envelope | [ITEM-DEFINITIONS-CORE.md § Definition Envelope](../../common/ITEM-DEFINITIONS-CORE.md#definition-envelope) | Definition payload structure | | Per-Item-Type Definitions | [ITEM-DEFINITIONS-CORE.md § Per-Item-Type Definitions](../../common/ITEM-DEFINITIONS-CORE.md#per-item-type-definitions) | Support matrix, decoded content, part paths — [REST specs](../../common/COMMON-CORE.md#item-creation), [CLI recipes](../../common/COMMON-CLI.md#item-crud-operations) | | Job Execution | [COMMON-CORE.md § Job Execution](../../common/COMMON-CORE.md#job-execution) || | Tool Selection Rationale | [COMMON-CLI.md § Tool Selection Rationale](../../common/COMMON-CLI.md#tool-selection-rationale) || | Authentication Recipes | [COMMON-CLI.md § Authentication Recipes](../../common/COMMON-CLI.md#authentication-recipes) | `az login` flows and token acquisition | | Fabric Control-Plane API via `az rest` | [COMMON-CLI.md § Fabric Control-Plane API via az rest](../../common/COMMON-CLI.md#fabric-control-plane-api-via-az-rest) | **Always pass `--resource`**; includes pagination and LRO helpers | | Item CRUD Operations | [COMMON-CLI.md § Item CRUD Operations](../../common/COMMON-CLI.md#item-crud-operations) | Create, get/update definition, delete patterns | | Job Execution (CLI) | [COMMON-CLI.md § Job Execution](../../common/COMMON-CLI.md#job-execution) || | Gotchas & Troubleshooting (CLI-Specific) | [COMMON-CLI.md § Gotchas & Troubleshooting (CLI-Specific)](../../common/COMMON-CLI.md#gotchas--troubleshooting-cli-specific) | `az rest` audience, shell escaping, token expiry | | Quick Reference | [COMMON-CLI.md § Quick Reference](../../common/COMMON-CLI.md#quick-reference) | `az rest` template + token audience/tool matrix | | Authoring Capability Matrix | [DATAFLOWS-AUTHORING-CORE.md § Authoring Capability Matrix](../../common/DATAFLOWS-AUTHORING-CORE.md#authoring-capability-matrix) | Full CRUD lifecycle, parameterized refresh, Git CI/CD | | Dataflow Definition Structure | [DATAFLOWS-AUTHORING-CORE.md § Dataflow Definition Structure](../../common/DATAFLOWS-AUTHORING-CORE.md#dataflow-definition-structure) | **Read first** — 3-part definition: queryMetadata.json, mashup.pq, .platform | | REST API Surface (Authoring) | [DATAFLOWS-AUTHORING-CORE.md § REST API Surface](../../common/DATAFLOWS-AUTHORING-CORE.md#rest-api-surface-authoring) | Create, getDefinition, updateDefinition, delete, run job | | Power Query M Code Structure | [DATAFLOWS-AUTHORING-CORE.md § Power Query M Code Structure](../../common/DATAFLOWS-AUTHORING-CORE.md#power-query-m-code-structure) | Section documents, multiple queries, parameters, fast copy | | Connection Model | [DATAFLOWS-AUTHORING-CORE.md § Connection Model](../../common/DATAFLOWS-AUTHORING-CORE.md#connection-model) | Connection kinds, path patterns, connectionId requirements | | Job Execution Patterns | [DATAFLOWS-AUTHORING-CORE.md § Job Execution Patterns](../../common/DATAFLOWS-AUTHORING-CORE.md#job-execution-patterns) | Trigger refresh, LRO polling, parameterized refresh | | ALM / Git Integration | [DATAFLOWS-AUTHORING-CORE.md § ALM / Git Integration](../../common/DATAFLOWS-AUTHORING-CORE.md#alm--git-integration) | Export/import, CI/CD pipeline, cross-environment overrides | | Gotchas and Troubleshooting | [DATAFLOWS-AUTHORING-CORE.md § Gotchas and Troubleshooting](../../common/DATAFLOWS-AUTHORING-CORE.md#gotchas-and-troubleshooting) | 15-row issue/cause/resolution table | | Quick Reference: Authoring Decision Guide | [DATAFLOWS-AUTHORING-CORE.md § Quick Reference](../../common/DATAFLOWS-AUTHORING-CORE.md#quick-reference-authoring-decision-guide) | Scenario → recommended approach lookup | | Core Authoring via CLI | [authoring-cli-quickref.md § Core Authoring via CLI](references/authoring-cli-quickref.md#core-authoring-via-cli) | Create, get/update definition, delete, refresh `az rest` one-liners | | Base64 Encoding Helpers | [authoring-cli-quickref.md § Base64 Encoding Helpers](references/authoring-cli-quickref.md#base64-encoding-helpers) | Encode/decode definition parts in bash and PowerShell | | Definition Manipulation | [authoring-cli-quickref.md § Definition Manipulation Patterns](references/authoring-cli-quickref.md#definition-manipulation-patterns) | Read-modify-write workflow for definitions | | Bash Templates | [authoring-script-templates.md § Bash Templates](references/authoring-script-templates.md#bash-templates) | Create dataflow, read-modify-write, refresh with LRO, CI/CD export/import | | PowerShell Templates | [authoring-script-templates.md § PowerShell Templates](references/authoring-script-templates.md#powershell-templates) | Create dataflow, trigger refresh with polling | | Tool Stack | [SKILL.md § Tool Stack](#tool-stack) | `az` CLI + `jq` + `base64`; verify before first op | | Connection | [SKILL.md § Connection](#connection) | Workspace/dataflow ID discovery via REST | | Agentic Workflows | [SKILL.md § Agentic Workflows](#agentic-workflows) | **Start here** — discover→formulate→execute→verify | | Gotchas, Rules, Troubleshooting | [SKILL.md § Gotchas, Rules, Troubleshooting](#gotchas-rules-troubleshooting) | **MUST DO / AVOID / PREFER** checklists | | Agent Integration Notes | [authoring-cli-quickref.md § Agent Integration Notes](references/authoring-cli-quickref.md#agent-integration-notes) | Platform-specific tips (Copilot CLI, Claude Code) | --- ## Tool Stack | Tool | Role | Install | |---|---|---| | `az` CLI | **Primary**: Auth (`az login`), REST API calls (`az rest`), token acquisition. | Pre-installed in most dev environments | | `jq` | Parse and manipulate JSON responses and definition payloads. | Pre-installed or trivial | | `base64` | Encode/decode definition parts for the REST API. | Built into bash / `[Convert]::ToBase64String()` in PowerShell | | `curl` | Alternative to `az rest` when raw HTTP control is needed. | Pre-installed | > **Agent check** — verify before first operation: > ```bash > az --version 2>/dev/null || echo "INSTALL: https://aka.ms/install-azure-cli" > jq --version 2>/dev/null || echo "INSTALL: apt-get install jq OR brew install jq" > ``` --- ## Connection ### Discover Workspace and Dataflow IDs Per [COMMON-CLI.md](../../common/COMMON-CLI.md) Finding Workspaces and Items in Fabric: ```bash # List workspaces — find workspace ID by name az rest --method get \ --resource "https://api.fabric.microsoft.com" \ --url "https://api.fabric.microsoft.com/v1/workspaces" \ --query "value[?displayName=='MyWorkspace'].id" --output tsv # List dataflows in workspace — find dataflow ID by name WS_ID="<workspaceId>" az rest --method get \ --resource "https://api.fabric.microsoft.com" \ --url "https://api.fabric.microsoft.com/v1/workspaces/$WS_ID/dataflows" \ --query "value[?displayName=='MyDataflow'].id" --output tsv ``` ### Reusable Connection Variables ```bash WS_ID="<workspaceId>" DF_ID="<dataflowId>" API="https://api.fabric.microsoft.com/v1" RESOURCE="https://api.fabric.microsoft.com" ``` --- ## Agentic Workflows ### Discover → Formulate → Execute → Verify 1. **Discover** → List workspaces, list dataflows, get definition (decode to inspect M code and metadata). 2. **Formulate** → Create or modify M code (`mashup.pq`), update `queriesMetadata`, prepare all 3 definition parts. 3. **Execute** → Create or update dataflow via REST API, optionally trigger refresh. 4. **Verify** → Poll LRO for completion, get definition again to confirm changes, check refresh status. ```bash # 1. Discover — get current definition RESULT=$(az rest --method post \ --resource "$RESOURCE" \ --url "$API/workspaces/$WS_ID/dataflows/$DF_ID/getDefinition") # Decode mashup.pq to inspect current queries echo "$RESULT" | jq -r '.definition.parts[] | select(.path=="mashup.pq") | .payload' | base64 -d # 2. Formulate — edit M code, prepare new definition parts # (modify the decoded mashup, re-encode, build JSON payload) # 3. Execute — update definition az rest --method post \ --resource "$RESOURCE" \ --url "$API/workspaces/$WS_ID/dataflows/$DF_ID/updateDefinition?updateMetadata=true" \ --body @definition.json # 4. Verify — trigger refresh and poll LOCATION=$(az rest --method post \ --resource "$RESOURCE" \ --url "$API/workspaces/$WS_ID/items/$DF_ID/jobs/Execute/instances" \ --headers "Content-Length=0" \ --output none --include-response-headers 2>&1 | grep -i "^location:" | awk '{print $2}' | tr -d '\r') ``` --- ## Gotchas, Rules, Troubleshooting For full authoring gotchas: [DATAFLOWS-AUTHORING-CORE.md](../../common/DATAFLOWS-AUTHORING-CORE.md) Gotchas and Troubleshooting. For CLI-specific issues: [COMMON-CLI.md](../../common/COMMON-CLI.md) Gotchas & Troubleshooting (CLI-Specific). ### MUST DO - **`az login` first** — all `az rest` calls use the active session. No session → 401. - **Always `--resource "https://api.fabric.microsoft.com"`** — wrong audience = 401. - **Base64-encode all definition parts** — REST API expects `payloadType: "InlineBase64"`. - **Send all 3 parts in updateDefinition** — it's a full replacement, not incremental. - **Handle LRO polling** — `getDefinition`, `updateDefinition` (with def), and job execution return 202; poll via `Location` header. - **Verify workspace has capacity** — `GET /v1/workspaces/{id}` and check `capacityId`. - **Set `formatVersion` to `"202502"`** in `queryMetadata.json`. - **Set `loadEnabled: true`** for queries that should write output on refresh. ### AVOID - **Hardcoded GUIDs** — discover workspace/dataflow IDs via REST API (Connection section). - **Skipping base64 encoding** — payloads will be rejected or corrupted. - **Partial definition updates** — sending 1 or 2 parts silently drops queries. - **Creating dataflows with unbound connections** — queries referencing nonexistent `connectionId` will fail at refresh. - **Using GET for `getDefinition`** — it's a POST endpoint; GET returns 405. - **Constructing operation URLs manually** — always use the `Location` header from 202 responses. - **Duplicate `displayName`** values — not enforced but causes confusion. ### PREFER - **`az rest` over raw `curl`** — handles token acquisition and refresh automatically. - **`getDefinition` before `updateDefinition`** — read-modify-write prevents accidental data loss. - **Validate M code before creating dataflow** — syntax errors won't surface until refresh. - **`jq` for JSON manipulation** — build definition payloads programmatically. - **`"Automatic"` for parameter type in job execution** — lets engine infer from definition. - **`?updateMetadata=true`** on `updateDefinition` — ensures `.platform` changes (display name) are applied. - **Env vars** (`WS_ID`, `DF_ID`, `API`, `RESOURCE`) for script reuse. ### TROUBLESHOOTING | Symptom | Fix | |---|---| | 401 Unauthorized | Verify `az login` is active; check `--resource "https://api.fabric.microsoft.com"` |