// Reference for creating DataStage (batch) flows with the watsonx.data integration SDK. The SDK is verbose and permits exhaustive stage and property access: use it to author flows pyflow's compact DSL can't express, and to edit or optimize existing DataStage flows, including ones bootstrapped with pyflow.
| name | di-agent-flow-datastage |
| description | Reference for creating DataStage (batch) flows with the watsonx.data integration SDK. The SDK is verbose and permits exhaustive stage and property access: use it to author flows pyflow's compact DSL can't express, and to edit or optimize existing DataStage flows, including ones bootstrapped with pyflow. |
DataStage batch flows are authored by writing SDK-style Python code and submitting
via create_or_update_datastage_flow. Auth, project context, persistence, and
compilation are handled automatically.
recommend_datastage_stages(subutterances=[...])datastage_property_lookup(requests=[{"stage": "..."}])di-agent-knowledge-engine-datastage skill stages/di-agent-knowledge-engine-datastage skill optimization/overview.mdUse duplicate_asset(asset_id=..., asset_type="datastage_flow", project_id=...) as a safety net before making changes that could break a flow.
Editing an existing flow: Before modifying, duplicate the flow as a backup with a timestamped name ("{name} [backup YYYY-MM-DD]"). Work on the original — the backup is your rollback point. Keep at most one backup per flow to avoid clutter (delete old ones via delete_asset).
Iterating on a new flow: After a successful create_or_update_datastage_flow, if the user requests further edits, snapshot the working state first via duplicate_asset. This way you can restore if subsequent changes break it.
Restoration: Not a single tool call. Delete the broken flow, then duplicate the backup back to the original name — or simply point the user to the backup.
Caveats: Duplicating a flow does NOT duplicate any jobs or schedules attached to it. Be cautious when editing flows with active job runs. Cross-flow references (e.g. sub-flows) are not preserved in the backup.
flow_id, never by name — name returns incomplete stage datarecommend_datastage_stages / datastage_property_lookupflow_link from the tool result as a clickable link so the user can open the flow in the UIUser friendly name from datastage_property_lookup (e.g. "Number of rows (per partition)"), not the internal identifier (nrecs); show the internal name only in code blocks or when the user asks for the SDK propertyGenerates a Markdown bug report for an IBM watsonx.data integration session. User can invoke directly. The agent may propose it (and must wait for explicit acceptance) only after exhausting recovery options on a failure. Skip for non-watsonx.data integration sessions.
Complete API spec for pyflow, IBM's LLM-only Python DSL for authoring new batch or streaming flows on DataStage or StreamSets. The compact surface is built for high LLM authoring reliability — bootstrap here first, and fall back to the verbose engine-specific SDK only when pyflow cannot express a needed feature.
Q&A reference for the DataStage parallel engine — parallelism, partitioning theory, APT configuration files, concurrent job execution, restart/recovery, disk/resource tuning, dataset performance, flow optimization (partitioning/sorting/memory), and per-stage semantics. Use for conceptual engine questions and stage property lookups regardless of authoring tool.
Reference for StreamSets Data Collector engines and StreamSets environments — StreamSets environment configuration, StreamSets engine deployment (Docker/Podman), StreamSets job execution, StreamSets engine communication methods (tunneling/direct), StreamSets high availability and failover, StreamSets monitoring and resource management. Use ONLY when the user explicitly mentions StreamSets, Data Collector, or a StreamSets-specific concern.
Use when the user asks to generate a Substrait query plan, create a Substrait plan from natural language, convert a data query to Substrait JSON, write a Substrait DSL pipeline, translate a data request into a query plan, process a test entry from a JSONL dataset. Covers writing DSL code, calling MCP tools (compile_substrait_dsl, get_substrait_dsl_examples, load_test_entry), and self-correcting on compile errors. Trigger on: "generate substrait", "generate substrait dsl", "generate functional plan", "generate fp", "process entry", "run entry", "generate dsl".