alibabacloud-dataworks-datastudio-develop

name

alibabacloud-dataworks-datastudio-develop

description

DataWorks data development Skill. Create, configure, validate, deploy, update, move, and rename nodes and workflows. Manage components, file resources, and UDF functions. Covers 150+ node types: Shell, SQL, Python, DI, Flink, EMR, etc. Supports scheduled and manual workflow orchestration via aliyun CLI or Python SDK. WARNING: Supports mutating operations (Move, Rename) requiring explicit user confirmation. Delete operations are NOT supported by this skill. Triggers: DataWorks, data development nodes, workflows, FlowSpec, scheduling tasks, data integration, ETL pipelines, .spec.json. Also triggers for Alibaba Cloud data development, scheduling node configuration, FlowSpec format, or DI task orchestration.

DataWorks Data Development

❗ 5-SECOND SUMMARY — Read This First

Credentials: Run aliyun configure list first. The CLI is almost always pre-configured (STS token). Do NOT search for credential files — just check the CLI config.

Install plugin first: Run aliyun plugin install --names dataworks-public. All commands use plugin mode (kebab-case): aliyun dataworks-public create-node ...

APIs: create-workflow-definition → create-node (per node) → create-pipeline-run (deploy). For updates: update-node → create-pipeline-run. NEVER use deploy-file, submit-file, create-file, create-business.

FlowSpec: Copy the exact JSON from the "Quick Start" section below. Do NOT guess the format. version is "2.0.0", kind is "CycleWorkflow" or "Node". Common wrong values: apiVersion, type, Workflow, metadata — all wrong.

Publish after update: update-node → create-pipeline-run(type=Online) → poll get-pipeline-run → exec-pipeline-run-stage. deploy-file does NOT work.

⚡ MANDATORY: Read Before Any API Call

These absolute rules are NOT optional — violating ANY ONE means the task WILL FAIL:

Pre-check: Aliyun CLI >= 3.3.3 required

Run aliyun version to verify >= 3.3.3. If not installed or version too low, run curl -fsSL --connect-timeout 10 --max-time 120 https://aliyuncli.alicdn.com/setup.sh | bash to update, or see references/cli-installation-guide.md for installation instructions.

Aliyun CLI Setup (execute before first use):

# Plugin install (required)
aliyun plugin install --names dataworks-public
# Plugin update (run periodically)
aliyun plugin update --names dataworks-public
# AI-Mode: available commands are enable/disable/set-user-agent
# This skill DISABLES AI-Mode (exact parameter control required)
aliyun configure ai-mode disable
# Do NOT run: aliyun configure ai-mode enable
# Set user-agent for tracking
aliyun configure ai-mode set-user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop

Per-command UA flag (business commands only): every aliyun dataworks-public invocation must append --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop.

0. FIRST THING: Check CLI credentials. Before ANY aliyun command, run aliyun configure list. The CLI is typically pre-configured with valid STS token credentials — do NOT search for credential files (e.g., testconfig.json) before checking. If aliyun configure list shows a valid profile, use it directly. If multiple profiles exist, run aliyun configure switch --profile <name> to select the correct one. Priority: prefer a profile whose name contains dataworks (case-insensitive); otherwise use default. Do NOT skip this step. Do NOT run any aliyun dataworks-public command before switching. NEVER read/echo/print AK/SK values.

1. Install plugin before first use. Run aliyun plugin install --names dataworks-public. If already installed, run aliyun plugin update --names dataworks-public to ensure latest version. The plugin provides kebab-case commands (create-node, create-workflow-definition, etc.) which are the required invocation form.

2. ONLY use plugin mode (kebab-case). Every DataWorks API call must look like: aliyun dataworks-public create-node --project-id ... --spec '...'. Never use PascalCase RPC (CreateNode, CreateWorkflowDefinition) — always use plugin mode.

3. ONLY use these commands for create: create-workflow-definition → create-node (per node, with --container-id) → create-pipeline-run (to deploy).

4. ONLY use these commands for update: update-node (incremental, kind:Node) → create-pipeline-run (to deploy). Never use import-workflow-definition, deploy-file, or submit-file for updates or publishing. 4a. ONLY use these commands for deploy/publish: create-pipeline-run --type Online --object-ids <ID> → get-pipeline-run --id <PipelineRunId> (poll) → exec-pipeline-run-stage --id <PipelineRunId> --code <StageCode> (advance). NEVER use deploy-file, submit-file, , or — these are all legacy APIs that will fail. ⚠️ is (e.g. ), NOT a JSON array string. Wrapping it as will produce because the CLI passes the literal bracket text as the ID.

If you catch yourself typing ANY of these legacy commands, STOP IMMEDIATELY and re-read the Quick Start below: create-file, create-business, create-folder, --file-type, /bizroot, /workflowroot, deploy-file, submit-file, list-files, get-file, list-deployment-packages, get-deployment-package, create-deployment, approve-deployment, deploy-node, create-flow, create-file-depends, create-schedule

⚠️ FlowSpec Anti-Patterns

Agents commonly invent wrong FlowSpec fields. The correct format is shown in the Quick Start below.

❌ WRONG	✅ CORRECT	Notes
"apiVersion": "v1" or "apiVersion": "dataworks.aliyun.com/v1"	"version": "2.0.0"	FlowSpec uses version, not apiVersion
"kind": "Flow" or "kind": "Workflow"	"kind": "CycleWorkflow" (for workflows) or "kind": "Node" (for nodes)	Only Node, CycleWorkflow, ManualWorkflow are valid. "Workflow" alone is NOT valid
"metadata": {"name": "..."}	"spec": {"workflows": [{"name": "..."}]}	FlowSpec has no metadata field; name goes inside spec.workflows[0] or spec.nodes[0]
"type": "SHELL" (at node level)	"script": {"runtime": {"command": "DIDE_SHELL"}}	Node type goes in script.runtime.command
"schedule": {"cron": "..."}	"trigger": {"cron": "...", "type": "Scheduler"}	Scheduling uses trigger, not schedule
"script": {"content": "..."} without path	"script": {"path": "node_name", ...}	script.path is always required

🚀 Quick Start: End-to-End Workflow Creation

Complete working example — create a scheduled workflow with 2 dependent nodes:

# Step 1: Create the workflow container
aliyun dataworks-public create-workflow-definition \
  --project-id 585549 \
  --spec '{"version":"2.0.0","kind":"CycleWorkflow","spec":{"workflows":[{"name":"my_etl_workflow","script":{"path":"my_etl_workflow","runtime":{"command":"WORKFLOW"}}}]}}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop
# → Returns {"Id": "WORKFLOW_ID", ...}

# Step 2: Create upstream node (Shell) inside the workflow
# IMPORTANT: Before creating, verify output name "my_project.check_data" is not already used by another node (list-nodes)
aliyun dataworks-public create-node \
  --project-id 585549 \
  --scene DATAWORKS_PROJECT \
  --container-id WORKFLOW_ID \
  --spec '{"version":"2.0.0","kind":"Node","spec":{"nodes":[{"name":"check_data","id":"check_data","script":{"path":"check_data","runtime":{"command":"DIDE_SHELL"},"content":"#!/bin/bash\necho done"},"outputs":{"nodeOutputs":[{"data":"my_project.check_data","artifactType":"NodeOutput"}]}}]}}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop
# → Returns {"Id": "NODE_A_ID", ...}

# Step 3: Create downstream node (SQL) with dependency on upstream
# NOTE on dependencies: "nodeId" is the CURRENT node's name (self-reference), "output" is the UPSTREAM node's output
aliyun dataworks-public create-node \
  --project-id 585549 \
  --scene DATAWORKS_PROJECT \
  --container-id WORKFLOW_ID \
  --spec '{"version":"2.0.0","kind":"Node","spec":{"nodes":[{"name":"transform_data","id":"transform_data","script":{"path":"transform_data","runtime":{"command":"ODPS_SQL"},"content":"SELECT 1;"},"outputs":{"nodeOutputs":[{"data":"my_project.transform_data","artifactType":"NodeOutput"}]}}],"dependencies":[{"nodeId":"transform_data","depends":[{"type":"Normal","output":"my_project.check_data"}]}]}}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop

# Step 4: Set workflow schedule (daily at 00:30)
aliyun dataworks-public update-workflow-definition \
  --project-id 585549 \
  --id WORKFLOW_ID \
  --spec '{"version":"2.0.0","kind":"CycleWorkflow","spec":{"workflows":[{"name":"my_etl_workflow","script":{"path":"my_etl_workflow","runtime":{"command":"WORKFLOW"}},"trigger":{"cron":"00 30 00 * * ?","timezone":"Asia/Shanghai","type":"Scheduler"}}]}}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop

# Step 5: Deploy the workflow online (REQUIRED — workflow is not active until deployed)
aliyun dataworks-public create-pipeline-run \
  --project-id 585549 \
  --type Online --object-ids WORKFLOW_ID \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop
# → Returns {"Id": "PIPELINE_RUN_ID", ...}
# Then poll get-pipeline-run and advance stages with exec-pipeline-run-stage
# (see "Publishing and Deploying" section below for full polling flow)

Key pattern: create-workflow-definition → create-node (with --container-id + outputs.nodeOutputs) → update-workflow-definition (add trigger) → create-pipeline-run (deploy). Each node within a workflow MUST have outputs.nodeOutputs. The workflow is NOT active until deployed via create-pipeline-run.

Dependency wiring summary: In spec.dependencies, nodeId is the current node's own name (self-reference, NOT the upstream node), and depends[].output is the upstream node's output (projectIdentifier.upstream_node_name). The outputs.nodeOutputs[].data value of the upstream node and the depends[].output value of the downstream node must be character-for-character identical, otherwise the dependency silently fails.

Core Workflow

Environment Discovery (Required Before Creating)

Step 0 — Check CLI Credentials (MUST be the very first action): Run aliyun configure list. The CLI is almost always pre-configured with STS token credentials — do NOT claim "I have no credentials" or search for credential files before running this command. If the output shows a Valid profile, you have working credentials — proceed immediately. If multiple profiles exist, run aliyun configure switch --profile <name> (prefer dataworks-named profile, otherwise default). No aliyun dataworks-public command may run before this.

If credentials are empty or invalid, STOP HERE. Do not proceed with any API calls. Report the error to the user and instruct them to configure valid credentials outside this session (via aliyun configure or environment variables). Do not attempt workarounds such as writing config files manually or using placeholder values.

Before creating nodes or workflows, understand the project's existing environment. It is recommended to use a subagent to execute queries, returning only a summary to the main Agent to avoid raw data consuming too much context.

Subagent tasks:

1. Call list-workflow-definitions to get the workflow list
2. Call list-nodes to get the existing node list
3. Call list-data-sources AND list-compute-resources to get all available data sources and compute engine bindings (EMR, Hologres, StarRocks, etc.). list-compute-resources supplements list-data-sources which may not return compute-engine-type resources
4. Return a summary (do not return raw data):
   • Workflow inventory: name + number of contained nodes + type (scheduled/manual)
   • Existing nodes relevant to the current task: name + type + parent workflow
   • Available data sources + compute resources (name, type) — combine both lists
   • Suggested target workflow (if inferable from the task description)

Based on the summary, the main Agent decides: target workflow (existing or new, user decides), node naming (follow existing conventions), and dependencies (infer from SQL references and existing nodes).

Pre-creation conflict check (required, applies to all object types):

1. Name duplication check: Before creating any object, use the corresponding list command to check if an object with the same name already exists:
   • Workflow → list-workflow-definitions
   • Node → list-nodes (node names are globally unique within a project)
   • Resource → list-resources
   • Function → list-functions
   • Component → list-components
2. Handling existing objects: Inform the user and ask how to proceed (use existing / rename / update existing). Direct deletion of existing objects is prohibited
3. Output name conflict check (CRITICAL): A node's outputs.nodeOutputs[].data (format ${projectIdentifier}.NodeName) must be globally unique within the project, even across different workflows. Use list-nodes --name NodeName and inspect Outputs.NodeOutputs[].Data in the response to verify. If the output name conflicts with an existing node, the conflict must be resolved before creation — otherwise deployment will fail with "can not exported multiple nodes into the same output" (see troubleshooting.md #11b)

Certainty level determines interaction approach:

• Certain information → Use directly, do not ask the user
• Confident inference → Proceed, explain the reasoning in the output
• Uncertain information → Must ask the user

Creating Nodes

Unified workflow: Whether in OpenAPI Mode or Git Mode, generate the same local file structure.

⚠️ 必须先创建本地文件，再调用 API 无论使用 OpenAPI Mode 还是 Git Mode，都必须先在本地创建完整的节点文件夹（spec.json + 代码文件 + dataworks.properties），经过 build.py 合并和 validate.py 校验后，再调用 create-node/create-workflow-definition 命令。禁止跳过本地文件创建直接调用 API 构造 Spec。

Step 1: Create the Node Directory and Three Files

One folder = one node, containing three files:

my_node/
├── my_node.spec.json          # FlowSpec node definition
├── my_node.sql                # Code file (extension based on contentFormat)
└── dataworks.properties       # Runtime configuration (actual values)

spec.json — Copy the minimal Spec from references/nodetypes/{category}/{TYPE}.md, modify name and path, and use ${spec.xxx} placeholders to reference values from properties. If the user specifies trigger, dependencies, rerunTimes, etc., add them to the spec as well.

Code file — Determine the format (sql/shell/python/json/empty) based on the contentFormat in the node type documentation; determine the extension based on the extension field.

dataworks.properties — Fill in actual values:

projectIdentifier=<actual project identifier>
spec.datasource.name=<actual datasource name>
spec.runtimeResource.resourceGroup=<actual resource group identifier>

Do not fill in uncertain values — if omitted, the server automatically uses project defaults.

Reference examples: assets/templates/

Step 2: Submit

Default is OpenAPI (unless the user explicitly says "commit to Git"):

1. Use build.py to merge the three files into API input:

   python $SKILL/scripts/build.py ./my_node > /tmp/spec.json
   

   build.py does three things (no third-party dependencies; if errors occur, refer to the source code to execute manually):

   • Read dataworks.properties → replace ${spec.xxx} and ${projectIdentifier} placeholders in spec.json
   • Read the code file (including DI .json code files) → replace placeholders → embed into script.content
   • Output the merged complete JSON

2. Validate the spec before submission:

   python $SKILL/scripts/validate.py ./my_node
   

3. Pre-submission verification (MANDATORY) — Before calling create-node，必须通过 API 验证以下信息确实存在且正确：

   环境验证（首次提交前执行一次）：

   • runtimeResource.resourceGroup — 调用 list-resource-groups 确认资源组存在，使用返回的资源组 ID（如 Serverless_res_group_...），不要使用人类可读名称（如 cx_res_4）。如不确定，省略让服务端使用项目默认值
   • datasource — 计算引擎节点（ODPS_SQL、HOLOGRES_SQL 等）需要数据源。调用 list-data-sources 或 list-compute-resources 确认数据源名称和类型匹配。如不确定，省略让服务端使用项目默认值

   Spec 内容审查（每个节点提交前）：

   • script.runtime.command matches the intended node type (check references/nodetypes/{category}/{TYPE}.md)
   • script.content — For code nodes, confirm the merged spec contains non-empty code. For DI nodes specifically, script.content must be a valid DIJob JSON string with flat top-level keys type, version, steps, order, setting, extend — it is NOT the legacy DataX shape {"job":{"content":[{"reader":{"plugin":...}}]}}. If your generated content has a top-level wrapper or , you are using the wrong format from training memory; rewrite it to match and before calling

Git Mode (when the user explicitly requests): git add ./my_node && git commit, DataWorks automatically syncs and replaces placeholders

Minimum required fields (verified in practice, universal across all 130+ types):

• name — Node name
• id — Must be set equal to name. Ensures spec.dependencies[*].nodeId can match. Without explicit id, the API may silently drop dependencies
• script.path — Script path, must end with the node name; the server automatically prepends the workflow prefix
• script.runtime.command — Node type (e.g., ODPS_SQL, DIDE_SHELL)

Copyable minimal node Spec (Shell node example):

{"version":"2.0.0","kind":"Node","spec":{"nodes":[{
  "name":"my_shell_node","id":"my_shell_node",
  "script":{"path":"my_shell_node","runtime":{"command":"DIDE_SHELL"},"content":"#!/bin/bash\necho hello"}
}]}}

Other fields are not required; the server will automatically fill in project defaults:

• datasource, runtimeResource — If unsure, do not pass them; the server automatically binds project defaults
• trigger — If not passed, inherits the workflow schedule. Only pass when specified by the user
• dependencies, rerunTimes, etc. — Only pass when specified by the user
• outputs.nodeOutputs — Optional for standalone nodes; required for nodes within a workflow ({"data":"${projectIdentifier}.NodeName","artifactType":"NodeOutput"}), otherwise downstream dependencies silently fail. ⚠️ The output name (${projectIdentifier}.NodeName) must be globally unique within the project — if another node (even in a different workflow) already uses the same output name, deployment will fail with "can not exported multiple nodes into the same output". Always check with list-nodes before creating

Creating Workflows

⚠️ 工作流创建也必须先创建本地文件。 先为工作流内每个节点创建本地文件夹（spec.json + 代码文件 + dataworks.properties），验证通过后再依次调用 API。首次提交前，必须通过 list-resource-groups、list-data-sources 等命令确认资源组、数据源等环境信息存在且正确。目录结构详见 workflow-guide.md。

1. Create the workflow definition (minimal spec):

   {"version":"2.0.0","kind":"CycleWorkflow","spec":{"workflows":[{
     "name":"workflow_name","script":{"path":"workflow_name","runtime":{"command":"WORKFLOW"}}
   }]}}
   

   Call create-workflow-definition → returns WorkflowId
2. Create nodes in dependency order (each node passes --container-id WorkflowId)
   • Before each node: Check that ${projectIdentifier}.NodeName is not already used as an output by any existing node in the project (use list-nodes with --name and inspect Outputs.NodeOutputs[].Data). Duplicate output names cause deployment failure
   • Each node's spec must include outputs.nodeOutputs: {"data":"${projectIdentifier}.NodeName","artifactType":"NodeOutput"}
   • Downstream nodes declare dependencies in spec.dependencies: nodeId = current node's own name (self-reference), depends[].output = upstream node's output (see workflow-guide.md)
3. Verify dependencies (MANDATORY after all nodes created) — For each downstream node, call list-node-dependencies --id <NodeID>. If TotalCount is 0 but the node should have upstream dependencies, create-node silently dropped them. Fix immediately with update-node using spec.dependencies (see "Updating dependencies" below). Do NOT proceed to deploy until all dependencies are confirmed
4. Set the schedule — update-workflow-definition with trigger (if the user specified a schedule)
5. Deploy online (REQUIRED) — create-pipeline-run --type Online --object-ids <WorkflowId> → poll get-pipeline-run --id <PipelineRunId> → advance stages with exec-pipeline-run-stage --id <PipelineRunId> --code <StageCode>. A workflow is NOT active until deployed. Do not skip this step or tell the user to do it manually.

Detailed guide and copyable complete node Spec examples (including outputs and dependencies): references/workflow-guide.md

Updating Existing Nodes

Must use incremental updates — only pass the node id + fields to modify:

{"version":"2.0.0","kind":"Node","spec":{"nodes":[{
  "id":"NodeID",
  "script":{"content":"new code"}
}]}}

⚠️ Critical: update-node always uses "kind":"Node", even if the node belongs to a workflow. Do NOT use "kind":"CycleWorkflow" — that is only for workflow-level operations (update-workflow-definition).

Do not pass unchanged fields like datasource or runtimeResource (the server may have corrected values; passing them back can cause errors).

⚠️ Updating dependencies: To fix or change a node's dependencies via update-node, use spec.dependencies. Example:

{"version":"2.0.0","kind":"Node","spec":{"nodes":[{"id":"NodeID"}],"dependencies":[{"nodeId":"current_node_name","depends":[{"type":"Normal","output":"project.upstream_node"}]}]}}

Update + Republish Workflow

Complete end-to-end flow for modifying an existing node and deploying the change:

1. Find the node — list-nodes(--name xxx) → get Node ID
2. Update the node — update-node with incremental spec (kind:Node, only id + changed fields)
3. Publish — create-pipeline-run --type Online --object-ids <PublishObjectId> → poll get-pipeline-run --id <PipelineRunId> → advance stages with exec-pipeline-run-stage --id <PipelineRunId> --code <StageCode>. ⚠️ <PublishObjectId> selection rule: if the node sits inside a workflow (its path from get-node contains a /, e.g. wf_name/node_name), <PublishObjectId> MUST be the workflow ID, NOT the node ID — the API rejects intra-workflow node IDs with 未找到发布对象. Only standalone nodes (root path, no /) take their own ID.

# Step 1: Find the node
aliyun dataworks-public list-nodes --project-id $PID --name "my_node" --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop
# → Note the node Id from the response

# Step 2: Update (incremental — only id + changed fields)
aliyun dataworks-public update-node --project-id $PID --id $NODE_ID \
  --spec '{"version":"2.0.0","kind":"Node","spec":{"nodes":[{"id":"'$NODE_ID'","script":{"content":"SELECT 1;"}}]}}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop

# Step 3: Publish (see "Publishing and Deploying" below)
# IMPORTANT: if $NODE_ID's `path` from get-node contains a "/", it is in a workflow —
# replace $NODE_ID below with the workflow ID. Standalone nodes (root path) take their own ID.
aliyun dataworks-public create-pipeline-run --project-id $PID \
  --type Online --object-ids $NODE_ID \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop

Common wrong paths after update-node (all prohibited):

• ❌ deploy-file / submit-file — legacy APIs, will fail or behave unexpectedly
• ❌ import-workflow-definition — for initial bulk import only, not for updating or publishing
• ❌ list-files / get-file — legacy file model, use list-nodes / get-node instead
• ✅ create-pipeline-run → get-pipeline-run → exec-pipeline-run-stage

Publishing and Deploying

⚠️ NEVER use deploy-file, submit-file, list-deployment-packages, get-deployment-package, list-files, or get-file for deployment. These are all legacy APIs. Use ONLY: create-pipeline-run → get-pipeline-run → exec-pipeline-run-stage.

Publishing is an asynchronous multi-stage pipeline:

1. create-pipeline-run --type Online --object-ids <ID> → get PipelineRunId from Id field. ⚠️ --object-ids is space-separated bare IDs, NOT a JSON array string. Pass the ID literally: --object-ids 7567482277219412494. Wrapping it as '["..."]' causes the CLI to send the bracketed text as the literal ID, producing 未找到发布对象: [["..."]]. The API only publishes the first ID and its child entities — for independent objects, run separate create-pipeline-run calls. For nodes inside a workflow: pass the workflow ID, NOT the node ID (intra-workflow node IDs are rejected with the same 未找到发布对象 error)
2. Poll get-pipeline-run --id <PipelineRunId> → check Pipeline.Status and Pipeline.Stages
3. When a Stage has Init status and all preceding Stages are Success → call exec-pipeline-run-stage --id <PipelineRunId> --code <Stage.Code> (the parameter is --id, NOT --pipeline-run-id) to advance
4. Until the Pipeline overall status becomes Success (deploy succeeded) — any other terminal status (Fail, Termination, Cancel) means the deploy did NOT succeed; see Rule #15 above

Key point: The Build stage runs automatically, but the Check and Deploy stages must be manually advanced. Detailed CLI examples, the full Status Decision Matrix, and polling scripts are in references/deploy-guide.md.

CLI Note: The aliyun CLI returns JSON with the top-level key Pipeline (not SDK's resp.body.pipeline); Stages are in Pipeline.Stages.

Common Node Types

Use Case	command	contentFormat	Extension	datasource
Shell script	DIDE_SHELL	shell	.sh	—
MaxCompute SQL	ODPS_SQL	sql	.sql	odps
Python script	PYTHON	python	.py	—
Offline data sync	DI	json	.json	—
Hologres SQL	HOLOGRES_SQL	sql	.sql	hologres
Flink streaming SQL	FLINK_SQL_STREAM	sql	.json	flink
Flink batch SQL	FLINK_SQL_BATCH	sql	.json	flink
EMR Hive	EMR_HIVE	sql	.sql	emr
EMR Spark SQL	EMR_SPARK_SQL	sql	.sql	emr
Serverless Spark SQL	SERVERLESS_SPARK_SQL	sql	.sql	emr
StarRocks SQL	StarRocks	sql	.sql	starrocks
ClickHouse SQL	CLICK_SQL	sql	.sql	clickhouse
Virtual node	VIRTUAL	empty	.vi	—

Complete list (130+ types): references/nodetypes/index.md (searchable by command name, description, and category, with links to detailed documentation for each type)

When you cannot find a node type:

1. Check references/nodetypes/index.md and match by keyword
2. Glob("**/{keyword}*.md", path="references/nodetypes") to locate the documentation directly
3. Use the get-node command to get the spec of a similar node from the live environment as a reference
4. If none of the above works → fall back to DIDE_SHELL and use command-line tools within the Shell to accomplish the task

Key Constraints

1. script.path is required: Script path, must end with the node name. When creating, you can pass just the node name; the server automatically prepends the workflow prefix
2. Dependencies are configured via spec.dependencies: In spec.dependencies, nodeId is a self-reference — it must be the current node's own name (the node being created), NOT the upstream node. depends[].output is the upstream node's output (${projectIdentifier}.UpstreamNodeName). The upstream's outputs.nodeOutputs[].data and downstream's depends[].output must be character-for-character identical. Upstream nodes must declare outputs.nodeOutputs. ⚠️ Output names (${projectIdentifier}.NodeName) must be globally unique within the project — duplicates cause deployment failure
3. Immutable properties: A node's command (node type) cannot be changed after creation; if incorrect, inform the user and suggest creating a new node with the correct type
4. Updates must be incremental: Only pass id + fields to modify; do not pass unchanged fields like datasource/runtimeResource
5. datasource.type may be corrected by the server: e.g., flink → flink_serverless; use the generic type when creating
6. Nodes can exist independently: Nodes can be created at the root level (without passing --container-id) or belong to a workflow (pass --container-id WorkflowId). Whether to place in a workflow is the user's decision
7. Workflow command is always WORKFLOW: script.runtime.command must be "WORKFLOW"
8. Deletion is not supported by this skill: This skill does not provide any delete operations. When creation or publishing fails, never attempt to "fix" the problem by deleting existing objects. Correct approach: diagnose the failure cause → inform the user of the specific conflict → let the user decide how to handle it (rename / update existing)
9. Name conflict check is required before creation: Before calling any Create API, use the corresponding List API to confirm the name is not duplicated (see "Environment Discovery"). Name conflicts will cause creation failure; duplicate node output names () will cause dependency errors or publishing failure

API Quick Reference

API Version: All APIs listed below are DataWorks 2024-05-18 version. Only use the APIs listed in the table below; do not search for or use other DataWorks APIs.

Business call format (plugin mode, UA flag required on every call): aliyun dataworks-public {kebab-case-command} --parameter --user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-datastudio-develop — for plugin installation and global UA configuration see the "Aliyun CLI Setup" section above.

Detailed parameters and code templates for each API are in references/api/{APIName}.md. If a call returns an error, you can get the latest definition from https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/{APIName}/api.json.

Components

Command	Description
create-component	Create a component
get-component	Get component details
update-component	Update a component
list-components	List components

Nodes

Command	Description
create-node	Create a data development node. project-id + scene + spec, optional container-id
update-node	Update node information. Incremental update, only pass id + fields to change
move-node	Move a node to a specified path
rename-node	Rename a node
get-node	Get node details, returns the complete spec
list-nodes	List nodes, supports filtering by workflow
list-node-dependencies	List a node's dependency nodes

Workflow Definitions

Command	Description
create-workflow-definition	Create a workflow. project-id + spec
import-workflow-definition	Import a workflow (initial bulk import ONLY — do NOT use for updates or publishing; use update-node + create-pipeline-run instead)
update-workflow-definition	Update workflow information, incremental update
move-workflow-definition	Move a workflow to a target path
rename-workflow-definition	Rename a workflow
get-workflow-definition	Get workflow details
list-workflow-definitions	List workflows, filter by type

Resources

Command	Description
create-resource	Create a file resource
update-resource	Update file resource information, incremental update
move-resource	Move a file resource to a specified directory
rename-resource	Rename a file resource
get-resource	Get file resource details
list-resources	List file resources

Functions

Command	Description
create-function	Create a UDF function
update-function	Update UDF function information, incremental update
move-function	Move a function to a target path
rename-function	Rename a function
get-function	Get function details
list-functions	List functions

Publishing Pipeline

Command	Description
create-pipeline-run	Create a publishing pipeline. type=Online/Offline
exec-pipeline-run-stage	Execute a specified stage of the publishing pipeline, async requires polling
get-pipeline-run	Get publishing pipeline details, returns Stages status
list-pipeline-runs	List publishing pipelines
list-pipeline-run-items	Get publishing content

Auxiliary Queries

Command	Description
get-project	Get projectIdentifier by id
list-data-sources	List data sources
list-compute-resources	List compute engine bindings (EMR, Hologres, StarRocks, etc.) — supplements list-data-sources
list-resource-groups	List resource groups

Reference Documentation

Scenario	Document
Complete list of APIs and CLI commands	references/related-apis.md
RAM permission policy configuration	references/ram-policies.md
Operation verification methods	references/verification-method.md
Acceptance criteria and test cases	references/acceptance-criteria.md
CLI installation and configuration guide	references/cli-installation-guide.md
Node type index (130+ types)	references/nodetypes/index.md
FlowSpec field reference	references/flowspec-guide.md
Workflow development	references/workflow-guide.md
Scheduling configuration	references/scheduling-guide.md
Publishing and unpublishing	references/deploy-guide.md
DI data integration