ae-dataops

name	ae-dataops
version	2.0.0
description	AE Data Development and Operations: Data warehouse management, flow orchestration, IDE queries, and data integration
metadata	{"requires":{"bins":["ae-cli"]}}

ae-dataops

CRITICAL - This skill is self-contained. Use the Global AE CLI Rules below; do not require a separate shared skill for DataOps-side tasks.

The AE Data Development and Operations domain provides capabilities for data warehouse management, flow orchestration, IDE SQL queries, data integration, operations and backfill management, including the following subcommands:

Subcommand	Responsibility	Corresponding Scenario Skill
dataops_repo	Space discovery	—
dataops_datatable	Data table and view management	dataops-table
dataops_flow	Flow creation and orchestration	dataops-flow-create
dataops_flow	Flow execution and monitoring	dataops-flow-monitor
dataops_operations	Operations instance search, details, and task logs	dataops-flow-monitor
dataops_ide	Data exploration and SQL queries	dataops-query
dataops_integration	Datasource and data integration	dataops-integration

---

Global AE CLI Rules

AE CLI (ae-cli) is the command-line tool for the AE / TE / ThinkingEngine analysis platform. For AE analysis-side requests, prefer ae-cli and this skill's reference docs over model memory.

Global parameters:

Parameter	Description
--format <json|table>	Output format. Default is JSON.
--jq <expr>	jq filter expression for JSON output.
--host <url>	Override the active AE host. Available on every command and may be placed after the subcommand, e.g. ae-cli dataops_ide +<command> --host <url>.

Output and errors:

• Successful commands return machine-readable JSON by default. Envelope may include optional _notice.host_compat.
• Failed commands return { "ok": false, "error": { "type": "...", "message": "...", "hint": "..." } } and exit non-zero.
• CRITICAL — Host compat (do this first): After each ae-cli run, check stderr and _notice.host_compat. If either is present, open the user reply with a short ⚠️ version warning and quote the npm i -g / npx skills add (or update-cluster) lines verbatim, then present the business result. Soft tip; ok: true can still carry the notice.

Safety constraints:

• Read commands can execute directly after required IDs and references are verified.
• Ordinary write commands execute without --yes; use --yes only for a high-risk-write command after explicit user confirmation.
• Never invent command names, flags, JSON payloads, project_id, resource IDs, field names, event names, property names, metric definitions, or date formats. Read the matching command reference and discover real project metadata first.
• NEVER fabricate or guess resource names (reports, dashboards, events, properties, metrics, clusters, tags, alerts). Always use list commands to discover real resources first. If a resource is not found after fuzzy search and full list fallback, explicitly tell the user "resource not found" and stop - do not proceed with fabricated names.

Domains for DataOps: dataops_repo, dataops_datatable, dataops_flow, dataops_operations, dataops_ide, dataops_integration

---

Core Concepts and Rules

You must understand the following key concepts before use, otherwise errors are highly likely.

ID System

ID	Source	Usage Scope
executeId	Returned by dataops_flow +execute_flow	Early stop handle before the scheduler flowInstanceId is available
flowInstanceId	Returned by dataops_operations +search_flow_instances	Operations perspective instance inspection and stop

Environment and Defaults

Scenario	Default Environment	Description
Most flow/ide/datatable commands	DEV	Development environment
dataops_operations +search_flow_instances	Operations instance search	Filter by keyword, execution date, status, and paging
dataops_operations +get_flow_instance_detail	Instance detail	Inspect one instance DAG and task statuses
dataops_operations +get_task_instance_detail	Task detail/logs	Inspect one task and include logs only when needed
dataops_operations +stop_flow_instance	Instance stop	Stop by exactly one of executeId or flowInstanceId

Schema Naming Rules

• DEV environment: ws_${spaceCode}_dev
• PROD environment: ws_${spaceCode}_product

Responsibility Boundaries

Operation	Correct Tool	Prohibited
Execute SELECT queries	dataops_ide	—
Create/modify/delete data tables (DDL)	dataops_datatable	dataops_ide

Flow Lifecycle

Create DEV Flow → Create/Update DEV SQL, Integration, or Workflow Instance Check Tasks → Configure Dependencies/Schedule → Preview Release → Release to PROD → PROD Manual Execution / Operations Troubleshooting

CRON Format (6 fields)

second minute hour day month weekday — Note: one more "second" field than standard 5-field format.

• 0 0 2 * * ? — Daily at 2 AM
• 0 0 */4 * * ? — Every 4 hours
• 0 30 8 * * 1-5 — Weekdays at 8:30

Preset Repository vs Non-Preset Repository

• Preset Repository (te_etl): datasourceId is te_etl@TASK_ENGINE_TRINO, database field is empty, requires gatewayConfig
• Non-Preset Repository: datasourceId is specific datasource ID, database field is required

---

Scenario Routing

Choose the appropriate scenario skill based on user intent to get complete step-by-step workflow guidance.

User Intent	Trigger Skill	Keywords
Create flow, add nodes, configure schedule, release	dataops-flow-create	create flow, new workflow, configure schedule, add task node, release, cron, scheduled execution
View execution status, troubleshoot failures, view logs	dataops-flow-monitor	execute flow, running instance, monitor, logs, stop, DAG, troubleshoot
Search operation instances across a space	dataops-flow-monitor	operations instance, flow instance search, status statistics, owner statistics
Create datasource, configure sync solution, execute sync	dataops-integration	datasource, sync, integration, field mapping, data ingestion, MySQL, ClickHouse, DatabricksJdbc
Browse metadata, search tables, execute SQL queries	dataops-query	query, SQL, data exploration, search tables, view table structure, IDE, catalog, select
Create tables and views	dataops-table	create table, table creation, view, data dictionary, table details, DDL

---

1. Space Discovery

dataops_repo exposes only one read command. Use it to discover a valid spaceCode before calling DataOps commands that require one. It returns createTime, spaceCode, and spaceDisplayName.

• If the user already provided a trusted spaceCode, reuse it.
• If spaceCode is unknown, run +list_spaces first.
• If exactly one space is returned, use its spaceCode.
• If multiple spaces are returned and the user intent does not identify one, ask the user which space to use. Do not guess.

# List spaces accessible to the current user
ae-cli dataops_repo +list_spaces

---

2. Data Table and View Management

Detailed workflow, command flags, examples, and parameter notes live in references/dataops-table.md.

Key constraints:

• Start with dataops_datatable +dict_search_tables for visible DataOps catalog discovery.
• Use dataops_ide +search_tables only for raw engine metadata, and dataops_ide +ide_list_tables only for known catalog/schema browsing.
• Create tables/views with dataops_datatable, not dataops_ide; creation is DEV-only and must be published with +publish_entity.
• DDL follows Trino syntax; current-space view DDL should keep the literal ${env} placeholder.

---

3. Flow Orchestration

Flow orchestration is divided into two scenario skills: creation and configuration and execution and monitoring.

Lifecycle: DEV configuration and preview → Release to PROD → PROD manual execution and operations troubleshooting

Detailed creation/configuration commands live in references/dataops-flow-create.md. Detailed execution, monitoring, operation instance, task log, and stop commands live in references/dataops-flow-monitor.md.

Key constraints:

• Create and update tasks in DEV, preview/release before PROD execution.
• +execute_flow always runs PROD; it returns executeId for early stop.
• Prefer flowInstanceId from operations search for stable inspection and troubleshooting.
• Reference workspace parameters in task SQL as ${paramKey}.

---

4. IDE SQL Queries

Detailed metadata browsing, SQL query, async download, and cancel workflows live in references/dataops-query.md.

Key constraints:

• IDE is query-only; create/modify/delete tables with dataops_datatable.
• Prefer dataops_datatable +dict_search_tables for table discovery unless raw engine metadata or schema browsing is required.
• SQL query submission creates a download task; rows are not returned inline.

---

5. Data Integration

Detailed datasource, metadata browsing, sync solution, execution, and monitoring workflows live in references/dataops-integration.md.

Key constraints:

• Generate sourceConfig, sinkConfig, channelConfig, and fieldsMapping from the reference templates; do not invent keys.
• MySQL Source read partitioning uses sourceConfig.splitColumn; fieldsMapping.shardingKey is column metadata and must not be used for it.
• +save_sync_solution is not a partial patch: call +get_sync_detail --withParams true first, then submit complete configs. syncName is accepted for compatibility but ignored.
• Preset repository sync uses te_etl@TASK_ENGINE_TRINO and requires gateway configuration.
• Use +list_sync_runs to get taskId before stopping a running sync.

---

Reference Documentation

For detailed command flags and usage, please refer to the command documentation in the references/ directory.