di-agent-flow-datastage

name	di-agent-flow-datastage
description	Reference for the verbose watsonx.data integration SDK for DataStage (batch) flows, with exhaustive stage and property access. New flows can only be created with pyflow's compact DSL (`di-agent-flow-pyflow`); use this SDK to edit existing flows in place — for what pyflow can't express.

Edit DataStage Flows

Existing DataStage batch flows are edited by writing SDK-style Python code and submitting via update_datastage_flow. Auth, project context, persistence, and compilation are handled automatically.

New flows should be created by pyflow (di-agent-flow-pyflow / create_pyflow) in almost every case and later updated with SDK code (update_datastage_flow) to add engine-specific details. However, if the intended flow contains stages that are unsupported by Pyflow, lacks data sources upfront, or create_pyflow fails repeatedly, you may create the flow with SDK code using the create_datastage_flow tool. Only create the flow with SDK code if pyflow cannot.

Where to look

• SDK conventions (method signatures, stage config, link schemas, column types, connection binding, key rules, full example) → references/sdk-conventions.md
• Stage selection → recommend_datastage_stages(subutterances=[...])
• Stage property names and accepted values → datastage_property_lookup(requests=[{"stage": "..."}])
• Per-stage deep-dive → di-agent-knowledge-engine-datastage skill stages/
• Flow optimization → di-agent-knowledge-engine-datastage skill optimization/overview.md
• Custom stages (C/C++, Java) → BuildopStage.md, JavaIntegrationStage.md
• Transformer stage expressions → di-agent-knowledge-engine-datastage skill resource stages/TransformerStageFunctions/TransformerStageFunctionsOverview.md

Versioning

Use duplicate_asset(asset_id=..., asset_type="datastage_flow", project_id=...) as a safety net before making changes that could break a flow.

Editing an existing flow: In-place SDK edit is the right tool when the change is confined to expressions or stage properties on structure that already exists — retrieve the flow with retrieve_datastage_flow_code, change that SDK, and resubmit via update_datastage_flow. You are perturbing validated, already-wired code, so this is reliable, lossless, and cheap.

Edit the retrieved code — never hand-author a fragment. Your update_datastage_flow submission must be the complete SDK returned by retrieve_datastage_flow_code with your additions spliced in, not a standalone snippet. Every stage, link, and schema you reference must be defined as a variable earlier in the same submission (the retrieved code already defines them — derive_seg, veh_in, fp, etc. — so reuse those variable names). The flattened SDK grammar has no way to fetch existing elements: flow.links["x"], flow.stages["y"], and link.schema are all rejected as "Unsupported statement type" / "variable does not exist". To add a column to an existing link, edit that link's existing create_schema()/add_field(...) lines in the retrieved code; to add structure, append new flow.add_stage(...) / stage.connect_output_to(...) / link.create_schema() statements that reference the already-defined variables. If a submission fails validation, fix the whole resubmitted code; do not re-fetch elements or re-author from scratch.

If the change instead adds or alters structure — a new source, a join, a different shape — do NOT hand-author it here from scratch; that is the SDK's least reliable path. Bootstrap the new structure in pyflow (di-agent-flow-pyflow), then precision-edit the generated SDK to add anything pyflow can't express — an unsupported function, or an extra stage spliced in. If the core shape has no faithful pyflow form, bootstrap the closest shape pyflow can produce, then reshape it here via update_datastage_flow.

Backups are optional and cost tool calls — only duplicate as a rollback point when the user values the existing flow as something to protect (an established production flow they asked you to change carefully). Skip the backup for a one-shot "finish it / fix it and run it" request on a partial or scratch flow: just retrieve, edit, resubmit, and run. When you do back up, use a timestamped name ("{name} [backup YYYY-MM-DD]"), work on the original, and keep at most one backup per flow (delete old ones via delete_asset). Do not spend calls duplicating, deleting, or re-duplicating while you are still trying to get a single edit to validate.

Iterating on a new flow: Once a freshly bootstrapped flow reaches a working state, if the user requests further edits, snapshot the working state first via duplicate_asset. This way you can restore if subsequent changes break it.

Restoration: Not a single tool call. Delete the broken flow, then duplicate the backup back to the original name — or simply point the user to the backup.

Caveats: Duplicating a flow does NOT duplicate any jobs or schedules attached to it. Be cautious when editing flows with active job runs. Cross-flow references (e.g. sub-flows) are not preserved in the backup.

Guardrails

• Fetch flows by flow_id, never by name — name returns incomplete stage data
• Never guess stage types, property names, or enum values — use recommend_datastage_stages / datastage_property_lookup
• Bundle all changes into a single submission — successive submissions overwrite each other
• update_datastage_flow never creates — it errors if the named flow doesn't exist; verify the name via list_datastage_flows and create the flow first
• After a successful update, surface the returned flow_link from the tool result as a clickable link so the user can open the flow in the UI
• Stage property names in prose — use the User friendly name from datastage_property_lookup (e.g. "Number of rows (per partition)"), not the internal identifier (nrecs); show the internal name only in code blocks or when the user asks for the SDK property