| name | alibabacloud-dataworks-infra-manage |
| description | DataWorks Infrastructure Management: Create and query operations for Data Sources (50 types), Compute Resources, and Serverless Resource Groups, plus connectivity testing and resource group binding/unbinding.
Uses aliyun CLI to call dataworks-public OpenAPI (2024-05-18).
Trigger keywords: DataWorks data source, compute resource, resource group, datasource, data source, compute resource, resource group,
mysql/hologres/maxcompute data source, holo/mc/flink resource, Serverless resource group, DataWorks infra, create/list datasource,
DW environment config, infrastructure initialization, connect database to DataWorks, database connection failure, configure holo/mc resource.
Not triggered: data development tasks, scheduling configuration, MaxCompute table management, data integration tasks, ECS/RDS/OSS operations, workspace member management, data quality monitoring, data lineage, data preview.
|
DataWorks Infrastructure Management
Unified management of Data Sources, Compute Resources, and Resource Groups in Alibaba Cloud DataWorks workspaces, supporting create and query operations.
Architecture
DataWorks
├── Workspaces ─── Query and search workspaces
│ ├── Data Sources ─── 50 types: MySQL, Hologres, MaxCompute, ...
│ └── Compute Resources ─── Hologres, MaxCompute, Flink, Spark
└── Resource Groups ─── Serverless resource group management (cross-workspace)
Dependencies:
Workspace ◀── Data Sources, Compute Resources (must belong to a workspace)
Workspace ◀── Resource Groups (associated via binding; one resource group can bind to multiple workspaces)
Connectivity Test ──depends on──▶ Resource Group (must be bound to the workspace of the data source)
Standard Mode ──requires──▶ Dev (Development) + Prod (Production) dual data sources and compute resources
Global Rules
Prerequisites
- Aliyun CLI >= 3.3.1:
aliyun version (Installation guide: references/cli-installation-guide.md)
- First-time use:
aliyun configure set --auto-plugin-install true
- jq (required for resource group operations):
which jq
- Credential status:
aliyun configure list, verify valid credentials exist
- DataWorks edition: Basic edition or above required
Security Rules: DO NOT read/print/echo AK/SK values, DO NOT let users input AK/SK directly, ONLY use aliyun configure list to check credential status.
Command Formatting
- User-Agent (mandatory): All
aliyun CLI commands must include the --user-agent AlibabaCloud-Agent-Skills parameter to identify the source.
- Single-line commands: When executing Bash commands, must construct as a single-line string; do not use
\ for line breaks.
- jq step-by-step execution: First execute the
aliyun command to get JSON, then format with jq (to avoid multi-line security prompts).
- Endpoint mandatory: When specifying the
--region parameter, you must also add --endpoint dataworks.<REGION_ID>.aliyuncs.com. Not needed when --region is not specified.
Parameter Confirmation
Before executing any command, all user-customizable parameters must be confirmed by the user. Do not assume or use default values.
Exception: When the user has explicitly specified parameter values in the conversation, use them directly without re-confirmation.
Resource group related parameters (mandatory user selection): VPC, VSwitch, Resource Group ID (for binding/connectivity testing) — involve networking and billing, DO NOT auto-select; must display a list for the user to explicitly choose. Confirm even if there is only one option.
⚠️ Write API Execution Gate — MUST Check Before Every Write Operation
MANDATORY: Before calling any Write API (Create / Update / Delete / Bind / Unbind / Associate / Dissociate / Test), you MUST perform the following checks in order:
- Scan the entire SKILL.md for a Security Restriction or Disabled Operations notice that mentions the target API or module.
- If a restriction exists: BLOCK the operation immediately. Do NOT call the API. Respond to the user with:
- What operation is blocked and why
- The recommended alternative (e.g., use the DataWorks console, contact administrator)
- If no restriction exists: Proceed normally with parameter confirmation and execution.
This check is NOT optional. It applies to every single write operation without exception. Never skip this step.
Quick Reference — Blocked APIs in this skill:
| Module | Blocked APIs | Reason |
|---|
| Data Sources (Module 1) | UpdateDataSource, DeleteDataSource | Prevent accidental data loss, credential exposure, disruption of running tasks |
| Compute Resources (Module 2) | UpdateComputeResource, DeleteComputeResource | Prevent disruption of running development and scheduling tasks |
Allowed Write APIs: CreateDataSource, CreateComputeResource, CreateResourceGroup, AssociateProjectToResourceGroup, DissociateProjectFromResourceGroup, TestDataSourceConnectivity
RAM Permissions
All operations require dataworks:<APIAction> permissions. Creating resource groups additionally requires AliyunBSSOrderAccess and vpc:DescribeVpcs, vpc:DescribeVSwitches.
Full permission matrix: references/ram-policies.md
Quick Start: New Workspace Infrastructure Initialization
When the user is unsure about specific operations or has vague requirements, guide them through the following process:
- Environment check — Check CLI and credentials per Prerequisites
- Confirm workspace — Use
ListProjects to locate the workspace, GetProject to confirm the mode (Simple/Standard)
- Create compute resources — Guide engine type selection; the system will automatically create corresponding data sources. Standard Mode requires Dev+Prod pairs. Only pure storage-type data sources (MySQL, Kafka, etc.) need separate data source creation
- Create/bind resource groups — Query existing resource groups → let user select → bind. Guide creation when no resource groups are available
- Test connectivity — Test with bound resource groups; when all pass, inform "Infrastructure configuration complete"
After each step, proactively suggest the next action.
Next Step Guidance
After each write operation is completed and verified, proactively suggest follow-up actions:
| Completed Operation | Recommended Next Step |
|---|
| Create compute resource | Standard Mode: "Create the corresponding Dev resource?"; "Test connectivity?" |
| Create data source separately | "Test connectivity?"; Standard Mode: "Create Dev/Prod environment data sources?" |
| Create resource group | "Bind to a workspace?" |
| Bind resource group | "Test data source connectivity?" |
| Connectivity test passed | "Infrastructure is ready." |
| Connectivity test failed | Analyze the error cause, guide the fix |
| Unbind resource group | "Bind to another workspace?" |
Trigger Rules
Trigger scenarios: Data source create/query, compute resource create/query, resource group management, infrastructure initialization, colloquial aliases (DW database connection failure, configure holo/mc resources, create rg)
Not triggered: Data development tasks, scheduling configuration, MaxCompute table management, data integration tasks, ECS/RDS/OSS, workspace member management, data quality/lineage/preview. Standalone workspace queries are handled by the alibabacloud-dataworks-workspace-manage skill.
Interaction Flow
All operations follow: Identify module → Environment check → Collect parameters → Execute command → Verify result → Guide next step
Common aliases: DW=DataWorks, holo=Hologres, mc/MC/odps=MaxCompute, pg=PostgreSQL, rg=Resource Group, ds=Data Source, RDS=InstanceMode MySQL/PG/SQLServer, ADB=AnalyticDB
Naming suggestions: Data source {type}_{business}_{purpose}, Compute resource {type}_{business}, Resource group dw_{purpose}_rg_{env}
Module 0: Workspace Query
If the alibabacloud-dataworks-workspace-manage skill is available, prefer using it for workspace queries. The following is only a fallback.
aliyun dataworks-public ListProjects --user-agent AlibabaCloud-Agent-Skills --Status Available --PageSize 100
When searching by name, first get the full list then filter .PagingInfo.Projects[] by Name/DisplayName using jq.
Module 1: Data Source Management
Supports 50 data source types. See references/data-sources/README.md for details.
When do you need to create a data source separately? Creating a compute resource (Module 2) will automatically create the corresponding data source. Only pure storage-type databases (MySQL, PostgreSQL, Kafka, MongoDB, etc.) need separate creation.
Note: The following types do not currently support OpenAPI: hdfs
Connection modes: UrlMode (self-hosted databases, requires host/port) or InstanceMode (Alibaba Cloud managed instances, requires instanceId). When unsure, proactively ask the user. InstanceMode is preferred.
Instance query APIs: references/data-sources/instance-apis.md
⚠️ Security Restriction — See Write API Execution Gate (Global Rules) for mandatory pre-check
IMPORTANT: The DeleteDataSource and UpdateDataSource APIs are supported by the DataWorks service, but this skill has disabled modifying or deleting data sources for security reasons. Before attempting any write operation, the agent MUST check the Write API Execution Gate section.
If you need to modify or delete a data source, please use the DataWorks console directly or contact your administrator.
Connection Mode Quick Reference
ConnectionPropertiesMode selection determines required fields. InstanceMode is preferred when both are available.
| Mode | Types | Count |
|---|
| Both | mysql, postgresql, sqlserver, polardb, polardbo, polardb-x-2-0, apsaradb_for_oceanbase, drds, starrocks, analyticdb_for_mysql, analyticdb_for_postgresql, milvus, mongodb, redis, elasticsearch, kafka | 16 |
| InstanceMode only | hologres, dlf, opensearch | 3 |
| UrlMode only | oracle, mariadb, dm, db2, tidb, vertica, gbase8a, kingbasees, saphana, snowflake, maxcompute, hive, clickhouse, doris, selectdb, redshift, hbase, lindorm, oss, s3, ftp, ssh, tablestore, memcache, graph_database, datahub, loghub, restapi, salesforce, httpfile, bigquery | 31 |
hdfs — not supported via OpenAPI.
Full details: references/data-sources/README.md
Workspace Mode
Environment note: Prod (Production) is for production data processing; Dev (Development) is for development and debugging, physically isolated from production.
aliyun dataworks-public GetProject --user-agent AlibabaCloud-Agent-Skills --Id <PROJECT_ID> — check DevEnvironmentEnabled:
false → Simple Mode (1 data source, envType=Prod)true → Standard Mode (2 data sources, Dev + Prod, physically isolated)
Full mode comparison: references/data-sources/README.md
Task 1.1: Create Data Source (CreateDataSource)
aliyun dataworks-public CreateDataSource --user-agent AlibabaCloud-Agent-Skills [--region <REGION_ID> --endpoint dataworks.<REGION_ID>.aliyuncs.com] --ProjectId <PROJECT_ID> --Name <NAME> --Type <TYPE> --ConnectionPropertiesMode <UrlMode|InstanceMode> --ConnectionProperties '<JSON>' --Description "<DESC>"
ConnectionProperties common structure:
- UrlMode:
{"envType":"Prod","address":[{"host":"<IP>","port":<PORT>}],"database":"<DB>","username":"<USER>","password":"<PWD>"}
- InstanceMode:
{"envType":"Prod","instanceId":"<ID>","regionId":"<REGION>","database":"<DB>","username":"<USER>","password":"<PWD>"}
Special type structures (Oracle, MaxCompute, HBase, etc.): see references/data-sources/ per-type docs
Cross-account data source configuration: references/cross-account-datasources.md
Task 1.2: Get Data Source (GetDataSource)
aliyun dataworks-public GetDataSource --user-agent AlibabaCloud-Agent-Skills --Id <DATASOURCE_ID> [--region <REGION_ID> --endpoint dataworks.<REGION_ID>.aliyuncs.com]
Task 1.3: List Data Sources (ListDataSources)
aliyun dataworks-public ListDataSources --user-agent AlibabaCloud-Agent-Skills --ProjectId <PROJECT_ID> [--Types '["mysql"]'] [--EnvType <Dev|Prod>] [--PageNumber 1] [--PageSize 20]
Returns nested structure DataSources[].DataSource[]; Name/Type are in the outer layer, Id/Description in the inner layer.
Task 1.4: Test Connectivity (TestDataSourceConnectivity)
Process: Query resource group list → Let user select a resource group → Execute test.
aliyun dataworks-public ListResourceGroups --user-agent AlibabaCloud-Agent-Skills --ProjectId <PROJECT_ID>
aliyun dataworks-public TestDataSourceConnectivity --user-agent AlibabaCloud-Agent-Skills --DataSourceId <ID> --ProjectId <PROJECT_ID> --ResourceGroupId "<RG_ID>"
If error "resourceGroupId is not in the project", the resource group needs to be bound first (confirm with user, then execute AssociateProjectToResourceGroup).
Module 2: Compute Resource Management
Supports Hologres, MaxCompute, Flink, Spark, and other types. The system will automatically create corresponding data sources upon creation.
⚠️ Security Restriction — See Write API Execution Gate (Global Rules) for mandatory pre-check
IMPORTANT: For security reasons, this skill does NOT support modifying or deleting compute resources. Before attempting any write operation, the agent MUST check the Write API Execution Gate section. These operations are disabled to prevent:
- Accidental data loss or service interruption
- Disruption of running data development and scheduling tasks
- Unintended changes to production compute resource configurations
If you need to modify or delete a compute resource, please use the DataWorks console directly or contact your administrator.
authType Rules
- Dev environment:
authType is fixed as Executor
- Prod environment: Options are
PrimaryAccount (recommended), TaskOwner, SubAccount, RamRole. Default recommendation is PrimaryAccount unless user has special requirements
authType details and guidance: references/compute-resources/README.md
Type-Specific Notes
- Hologres: Only supports InstanceMode, requires
instanceId, securityProtocol
- MaxCompute: Only supports UrlMode, requires
project, endpointMode
Full ConnectionProperties examples: references/compute-resources/README.md
Task 2.1: Create Compute Resource (CreateComputeResource)
aliyun dataworks-public CreateComputeResource --user-agent AlibabaCloud-Agent-Skills [--region <REGION_ID> --endpoint dataworks.<REGION_ID>.aliyuncs.com] --ProjectId <PROJECT_ID> --Name <NAME> --Type <TYPE> --ConnectionPropertiesMode <InstanceMode|UrlMode> --ConnectionProperties '<JSON>' [--Description "<DESC>"]
After creation, use ListDataSources to verify the corresponding data source was auto-generated.
Task 2.2: Get Compute Resource (GetComputeResource)
aliyun dataworks-public GetComputeResource --user-agent AlibabaCloud-Agent-Skills --Id <ID> --ProjectId <PROJECT_ID>
Task 2.3: List Compute Resources (ListComputeResources)
aliyun dataworks-public ListComputeResources --user-agent AlibabaCloud-Agent-Skills --ProjectId <PROJECT_ID> [--Name <FILTER>] [--EnvType <Dev|Prod>] [--PageSize 20] [--SortBy CreateTime] [--Order Desc]
Returns nested structure ComputeResources[].ComputeResource[]; Name/Type are in the outer layer, Id in the inner layer.
Module 3: Resource Group Management
Manages the full lifecycle of Serverless resource groups.
Task 3.1: Create Resource Group (CreateResourceGroup)
Requires AliyunBSSOrderAccess permission.
Interaction flow (let user choose at each step, DO NOT auto-select):
- Query and select VPC:
aliyun vpc DescribeVpcs --user-agent AlibabaCloud-Agent-Skills --RegionId "<REGION_ID>" --PageSize 50
If the list is empty, guide the user to create a VPC; DO NOT auto-create.
- Query and select VSwitch:
aliyun vpc DescribeVSwitches --user-agent AlibabaCloud-Agent-Skills --RegionId "<REGION_ID>" --VpcId "<VPC_ID>" --PageSize 50
- Confirm name and specification → Execute creation:
aliyun dataworks-public CreateResourceGroup --user-agent AlibabaCloud-Agent-Skills [--region <REGION_ID> --endpoint dataworks.<REGION_ID>.aliyuncs.com] --Name "<NAME>" --PaymentType PostPaid --VpcId "<VPC_ID>" --VswitchId "<VSWITCH_ID>" --ClientToken "$(uuidgen 2>/dev/null || echo "token-$(date +%s)")" --Remark "Created by Agent"
After creation, poll GetResourceGroup until status becomes Normal (every 10 seconds, up to 10 minutes).
Task 3.2: Get Resource Group (GetResourceGroup)
aliyun dataworks-public GetResourceGroup --user-agent AlibabaCloud-Agent-Skills --Id "<ID>"
Task 3.3: List Resource Groups (ListResourceGroups)
aliyun dataworks-public ListResourceGroups --user-agent AlibabaCloud-Agent-Skills [--ProjectId <PROJECT_ID>] [--Statuses '["Normal"]'] --PageSize 100
Task 3.4: Bind Resource Group (AssociateProjectToResourceGroup)
Process: Query available resource groups → Display list for user to select → Bind after user confirms.
aliyun dataworks-public AssociateProjectToResourceGroup --user-agent AlibabaCloud-Agent-Skills --ResourceGroupId "<RG_ID>" --ProjectId "<PROJECT_ID>"
Task 3.5: Query Binding Relationships
aliyun dataworks-public ListResourceGroupAssociateProjects --user-agent AlibabaCloud-Agent-Skills --ResourceGroupId "<RG_ID>"
Task 3.6: Unbind Resource Group (DissociateProjectFromResourceGroup)
aliyun dataworks-public DissociateProjectFromResourceGroup --user-agent AlibabaCloud-Agent-Skills --ResourceGroupId "<RG_ID>" --ProjectId "<PROJECT_ID>"
Success Verification
After all write operations, use the corresponding Get/List command to verify the result.
Common Errors
| Error Code | Solution |
|---|
| Forbidden.Access / PermissionDenied | Check RAM permissions, see references/ram-policies.md |
| InvalidParameter | Check ConnectionProperties JSON and required parameters |
| EntityNotExists | Verify the ID and Region are correct |
| QuotaExceeded | Delete unused resources or request a quota increase |
| Duplicate* | Use a different name |
Region
Common: cn-hangzhou, cn-shanghai, cn-beijing, cn-shenzhen. Endpoint: dataworks.<region-id>.aliyuncs.com
Full list: references/related-apis.md
Best Practices
- Query before action — Confirm current state before create operations
- Manage by environment — Manage Dev and Prod resources separately
- Verify operations — Use Get/List to verify after each write operation
- Proactive guidance — Suggest the next step after each step completes
- Protect data sources and compute resources — Never modify or delete data sources or compute resources via this skill; use the DataWorks console for such operations
Reference Links
