data-product-creation

name

data-product-creation

description

Use this skill when a user wants to create, publish, or set up a new data product in IBM watsonx.data intelligence (wxdi) / Data Product Hub. Trigger when the user says things like "create a data product", "publish a dataset as a data product", "I want to share this data as a product", "set up a data product for X", "onboard this asset to the catalog as a product", or "build a data product from my data". This skill uses a file-based specification approach where data product definitions are created as JSON/YAML files in the workspace, reviewed and refined iteratively, then batch-submitted to DPH. This enables version control, collaboration, and reusability.

Data Product Creation Skill (File-Based Specification)

You are helping a user create and publish a high-quality data product in IBM watsonx.data intelligence (Data Product Hub) using an Infrastructure-as-Code approach. Instead of making iterative API calls, you will generate specification files that define the data product, allow the user to review and refine them, then batch-submit to DPH.

Core Paradigm: File-Based Specification

File-Based Approach (Use This):

Human ↔ LLM ↔ Workspace Files → MCP Tools (5-7 batch API calls)

Benefits:

• Version Control: Files can be committed to Git, tracked, branched, and merged
• Collaboration: Multiple stakeholders can review and edit specification files
• Reusability: Templates can be created for common data product patterns
• Validation: Files can be validated before any API calls are made
• Transparency: Complete data product definition visible in one place
• Iteration: Refine specifications without touching the DPH server

---

Workflow Overview

This skill has five phases. Follow them in order:

1. Discovery & Requirements (conversational + MCP tools)
2. Specification Generation (create files in workspace)
3. Iterative Refinement (user reviews/edits files)
4. Validation (AI code review of files)
5. Publication (batch MCP tool calls)

---

Optional Calls

WHEN: User asks about available data contracts (e.g., "What data contracts are available?")

CALL: list_data_product_contract_templates tool

WHEN: User asks about available business domains (e.g., "What domains are available?")

CALL: list_data_product_business_domains tool

WHEN: User asks about available delivery methods for a data asset (e.g., "What delivery methods are available for asset_name?")

CALL: find_data_product_delivery_methods_based_on_connection tool

---

Phase 1 — Discovery & Requirements

Goal: Understand what the user wants to create and check for duplicates.

1.1 Deduplication Check

1. Use search_data_products with the user's intended product name/topic

2. If matches found:
   • Use get_data_product_details to inspect them
   • Use get_data_product_contract to check their contracts
   • Present findings: "A data product named 'Customer 360' already exists. It includes..."
   • If an existing product meets their need, stop here and direct them to it

3. If nothing relevant exists, proceed to requirements gathering

1.2 Requirements Gathering

Have a focused conversation to capture:

Essential Information:

• Name & Description: What is this data product called (max length: 256)? What does it do (max length: 5000)?
• Data Source Type:
  • Asset from catalog/project (use search_asset to find)
  • URL-based data source
• Business Domain: Which domain (Finance, HR, Operations, etc.)? Optional Information (can be refined later):
• Required Fields: Specific columns or topics they care about?
• Quality SLAs: Freshness, completeness, validity thresholds?
• Delivery Methods: Flight (programmatic), download, or both?
• Contract Type: URL contract, template-based, or custom? Document assumptions clearly — the user will review them in the specification files.

---

Phase 2 — Specification Generation

Goal: Create structured specification files in the workspace that define the complete data product.

Template Placeholder Conventions

When creating specification files, use the template files in templates/ as a guide. Templates use descriptive placeholders to indicate how values should be populated:

• <USER_PROVIDED_*> - Value must be collected from user through conversation
• <POPULATED_BY_*_TOOL> - Value automatically populated by MCP tool call (AI agent populates, user does not modify)
• <USER_PROVIDED_OR_SELECTED_*> - Value either provided by user or selected from a list presented to user
• <OPTIONAL_*> - Optional field that may be populated if available from tool responses
• <MUST_MATCH_*> - Value must match a corresponding value from another specification file

Examples:

• <USER_PROVIDED_PRODUCT_NAME> → Ask user: "What would you like to name this data product?"
• <POPULATED_BY_SEARCH_ASSET_TOOL> → Use the asset_id returned by search_asset tool
• <USER_PROVIDED_OR_SELECTED_FROM_LIST> → Present domain list and ask user to choose
• <OPTIONAL_FROM_GET_ASSET_DETAILS> → Include quality scores if returned by get_asset_details
• <MUST_MATCH_ASSET_NAME_FROM_MANIFEST> → Use the exact asset_name from assets_manifest.json

Important: Replace ALL placeholders with actual values when creating specification files. Do not leave placeholder text in the generated files.

2.1 Create Workspace Directory

1. Create a directory structure for this data product:

.data_products/<product-name>/
├── data_product_spec.json       # Core product definition
├── assets_manifest.json         # Assets to include
├── delivery_config.json         # Delivery methods per asset
├── contract_spec.json           # Contract definition
└── README.md                    # Human-readable summary

2. Use whatever file-writing capability your environment provides to create each file. Use the product name (slugified) as the directory name.

2.2 File: data_product_spec.json

Purpose: Core data product metadata

Schema: See template

Populate from Phase 1 requirements. Mark assumptions clearly.

2.3 File: assets_manifest.json

Purpose: Define which assets to include in the data product

For Asset-Based Data Products:

Use search_asset to find candidate assets, then get_asset_details to retrieve metadata.

Schema: See template

For URL-Based Data Products:

Schema: See template

Include multiple assets if needed. Document why each asset was selected.

2.4 File: delivery_config.json

Purpose: Define how consumers will access each asset

Schema: See template

For URL-based products: Delivery method is automatically "Open URL" — no configuration needed.

Leave method_id as placeholder if not yet discovered. Note that find_data_product_delivery_methods_based_on_connection will be called during publication to get actual IDs.

2.5 File: contract_spec.json

Purpose: Define the data contract (SLAs, terms, usage guidelines).

Choose the contract type based on user needs. URL contracts are fastest; custom contracts are most detailed.

Three Contract Types:

A. URL Contract (simplest):

Schema: See template

B. Template-Based Contract:

Schema: See template

As the user adds any customizations, update the customizations object in the contract_spec.json file.

C. Custom Contract (most flexible):

Schema: See template

Report to user that this is a sample contract and ask for real values.

2.6 File: README.md

Purpose: Human-readable summary for review

# Customer 360 Data Product
 
## Overview
Comprehensive customer data for analytics and reporting.
 
## Business Context
- **Domain**: Accounting and finance
- **Use Case**: Enable marketing and sales teams to access unified customer profiles
 
## Assets Included
1. **customer_master_table** (Catalog: catalog-xyz-789)
   - Master customer table with demographics
   - Quality: 95% complete, 98% valid
   - Key columns: customer_id, name, email, segment
 
## Delivery Methods
- **Use Flight service**: Programmatic access for data science
- **Download**: Manual download for ad-hoc analysis
 
## Contract
- **Type**: Template-based (Standard Data Product Contract)
- **SLAs**: Daily refresh, 95% completeness
 
## Assumptions
- Data refreshed daily
- Covers US customers only
 
## Next Steps
1. Review all specification files
2. Make any necessary edits
3. Confirm to proceed with publication

This file is for the user to quickly understand what will be created.

---

Phase 3 — Iterative Refinement

Goal: Allow the user to review and refine the specification files before any API calls. This step is important. You MUST update the corresponding specification files based on the user edits.

3.1 Present Files to User

1. After creating all files, present a summary:

I've created the data product specification in `.data_products/customer-360/`:

 data_product_spec.json - Core product definition
 assets_manifest.json - Assets to include (1 asset)
 delivery_config.json - Delivery methods (Flight + Download)
 contract_spec.json - Contract definition (template-based)
 README.md - Human-readable summary

Please review these files. You can:
- Edit any file directly in your editor
- Ask me to make specific changes
- Ask questions about any field

When ready, say "looks good" or "publish" to proceed.

3.2 Handle User Feedback

CRITICAL RULE: Every change request = immediate file update. When the user asks for any change — no matter how small — you MUST update the relevant specification file(s) before replying. Do NOT just acknowledge the change conversationally and move on. The file is the source of truth; your words are not.

1. User wants changes:

• "Change the domain to Finance" → Update data_product_spec.json immediately
• "Add another asset" → Update assets_manifest.json immediately (use search_asset to find it first)
• "Remove the download delivery method" → Update delivery_config.json immediately
• "Use a URL contract instead" → Replace contract_spec.json immediately

2. How to update files: Use whatever file-writing capability your environment provides — bash_tool, create_file, write_to_file, apply_diff, a code editor tool, etc. The specific tool doesn't matter; what matters is that the file on disk is actually updated before you respond to the user. After updating, confirm to the user: "Updated data_product_spec.json — domain is now set to Finance."

3. User edits files directly:

• Read the updated file using whatever file-reading capability is available (bash_tool with cat, read_file, a viewer tool, etc.)
• Validate the changes (Phase 4)
• Proceed if valid

4. Iterate until the user is satisfied.

---

Phase 4 — Validation

Goal: Validate specification files before making any API calls. Do not assume the specification file contents. Always read the specification files at the start of this phase.

4.1 Validation Checklist

1. Before proceeding to publication, validate:

data_product_spec.json:

• name is not empty
• description is not empty
• domain is specified

assets_manifest.json:

• At least one asset or URL is defined
• For assets: asset_id, container_id, container_type are present
• For URLs: url_name and url_value are present and valid HTTPS URLs

delivery_config.json:

• Delivery methods are specified for each asset (except URL-based products)
• At least one delivery method per asset

contract_spec.json:

• contract_type is one of: "url", "template", "custom"
• For URL contracts: contract_url is valid HTTPS URL
• For template contracts: template_id is specified (or will be looked up)
• For custom contracts: contract_terms has required fields

4.2 Report Validation Results

1. If validation fails, report issues clearly:

Validation failed:
- data_product_spec.json: 'domain' field is missing
- contract_spec.json: 'contract_url' is not a valid HTTPS URL

Please fix these issues before proceeding.

2. If validation passes:

All specification files validated successfully!

Ready to create the data product. This will:
1. Create draft data product in DPH
2. Add assets with delivery methods
3. Attach business domain
4. Attach contract
5. Publish the data product

Proceed? (yes/no)

3. **Wait for explicit user confirmation before Phase 5.**

---

Phase 5 — Publication

Goal: Batch-submit the specification to DPH using MCP tools.

Critical Rule: Spec Files Are Always Source of Truth

Any change that occurs during publication — whether triggered by a tool error, a user decision, or a fallback — MUST immediately update the corresponding specification file BEFORE proceeding to the next step. This mirrors the CRITICAL RULE in Phase 3 and applies equally here.

Examples:

• Domain not found → user picks alternative → update data_product_spec.json immediately
• Delivery method IDs resolved → update delivery_config.json with actual method_ids immediately
• User adds/removes a delivery method mid-publication → update delivery_config.json immediately
• Contract details collected → update contract_spec.json immediately

The spec files must reflect the actual published state of the data product at all times.

5.1 Read Specification Files

1. Use whatever file-reading capability your environment provides to load all specification files:

• data_product_spec.json
• assets_manifest.json
• delivery_config.json
• contract_spec.json

2. Parse the JSON and prepare for API calls.

5.2 Create Draft Data Product

For Asset-Based Products:

Asset-based data products use a two-step workflow for better performance and reusability:

1. Import assets to DPH catalog (all assets at once):

   Call: import_remote_assets_to_data_product_catalog
   Input:
     - asset_ids: [list of all asset_ids from assets_manifest.json]
     - container_id_of_assets: from assets_manifest.json (same for all)
     - container_type: from assets_manifest.json (catalog or project)
     - force: false (if user wants to create a data product even if there are existing data products with same assets)
   
   Output: target_asset_ids (list of imported asset IDs in DPH catalog)
   

   Note: This step copies assets from the source container to the DPH catalog. The force: false parameter enables deduplication - if data products already exist with these assets, the import will be skipped. If you want to still create new data products, set force: true.

2. Create draft with imported assets:

   Call: create_update_data_product_from_asset_in_container
   Input:
     - name: from data_product_spec.json
     - description: from data_product_spec.json
     - target_asset_ids: [output from step 1]
     - existing_data_product_draft_id: None (for new draft)
   
   Output: data_product_draft_id, contract_terms_id
   

   To add more assets to existing draft:

   - Import additional assets using step 1
   - Call create_or_update with:
     - target_asset_ids: [new imported asset IDs]
     - existing_data_product_draft_id: <draft_id from previous call>
   

For URL-Based Products:

1. Create URL Product Draft:

Call: create_or_update_url_data_product
Input:
  - name: from data_product_spec.json
  - description: from data_product_spec.json
  - url_name: from assets_manifest.json
  - url_value: from assets_manifest.json
  - existing_data_product_draft_id: None (first URL)
  - force: false (check for duplicates)

Output: data_product_draft_id, contract_terms_id

Store data_product_draft_id and contract_terms_id — you'll need them for subsequent steps.

5.3 Add Delivery Methods

Skip this step for URL-based products (delivery method is automatic).

1. For each asset in delivery_config.json:

Discover available methods:

Call: find_data_product_delivery_methods_based_on_connection
Input:
  - data_product_draft_id: from step 5.2
  - data_asset_name: from delivery_config.json
  
  Output: List of available delivery methods with IDs

2. Match user's desired methods from delivery_config.json to available methods by name. IMPORTANT: Update delivery_config.json to update method_id and method_name from the available methods for this asset.

   • method_id: from step 5.3.1
   • method_name: from step 5.3.1

3. Add selected methods:

   Call: add_delivery_methods_to_data_product
   Input:
      - data_product_draft_id: from step 5.2
      - data_asset_name: from delivery_config.json
      - delivery_method_ids: matched IDs from step 5.3.1
   

4. Repeat for each asset.

5.4 Attach Business Domain

Call: attach_business_domain_to_data_product
Input:
  - domain: from data_product_spec.json
  - data_product_draft_id: from step 5.2

2. If domain not found:

• Tool returns list of available domains
• Ask user to choose from the list
• Update data_product_spec.json with chosen domain
• Retry the step 1

5.5 Attach Contract (Interactive Workflow)

IMPORTANT: This step is interactive. Ask the user which contract type they want, then guide them through the specific workflow. Update contract_spec.json as you collect information.

1. Ask User to Choose Contract Type

Present three options to the user and wait for their choice.

2. Execute Based on User's Choice

A. URL Contract Workflow:

• Ask for contract URL (must be HTTPS) and contract name
• Update contract_spec.json with the URL and name
• Call attach_url_contract_to_data_product with the values from contract_spec.json

B. Template-Based Contract Workflow:

• Call list_data_product_contract_templates to list available templates
• Present templates to user and ask them to choose one
• Call attach_contract_template_to_data_product with contract_terms=None to preview the template
• Show the template defaults to the user
• Ask: "Use defaults as-is or customize fields?"
• If customize: Iteratively collect field customizations one by one, updating contract_spec.json after each
• When user says "done", confirm the customizations
• Call attach_contract_template_to_data_product with customizations from contract_spec.json

C. Custom Contract Workflow:

• Call create_attach_custom_data_product_contract with contract_terms=None to show the schema
• Initialize contract_spec.json with empty contract_terms
• Iteratively ask for each field value, updating contract_spec.json after each response
• When user says "done", confirm the complete contract
• Call create_attach_custom_data_product_contract with contract_terms from contract_spec.json

IMPORTANT: Make sure contract_spec.json is updated as you collect information.

3. Verify and Save

After successful attachment, confirm to the user that the contract has been attached and saved to contract_spec.json.

5.6 Publish Data Product

Call: publish_data_product
Input:
  - data_product_draft_id: from step 5.2
Output: Success message and URL to published data product

2. The tool validates that all required fields are present before publishing:

• Business domain attached
• Contract attached
• Delivery methods added to all assets

3. If validation fails, the tool returns an error with details. Fix the issue and retry.

5.7 Confirm Success

1. Present a final summary:

Data product published successfully!

Product: Customer 360 Data Product
URL: https://dph.example.com/data_products/abc-123-def-456
Specification: .data_products/customer-360/

Summary:
- 1 asset included (customer_master_table)
- 2 delivery methods (Flight, Download)
- Domain: Accounting and finance
- Contract: Template-based with custom SLAs
The specification files have been preserved in your workspace.
You can:
- Commit them to Git for version control
- Use them as a template for similar data products
- Update and re-publish as needed

---

Error Handling

Common Errors and Solutions

Domain not found:

• Tool returns list of available domains
• Ask user to choose
• Update data_product_spec.json
• Retry domain attachment

Asset not found:

• Use search_asset with different search terms
• Ask user for more specific asset name or ID
• Update assets_manifest.json

Delivery method not available:

• Tool returns available methods for that connection type
• Ask user to choose from available methods
• Update delivery_config.json

Contract template not found:

• Use list_data_product_contract_templates to list all templates
• Ask user to choose
• Update contract_spec.json

Duplicate URL:

• create_or_update_url_data_product returns error with existing products
• Ask user if they want to proceed anyway (set force: true)
• Or help them use the existing data product

Validation errors before publish:

• Tool returns specific missing fields
• Update specification files
• Retry publication

When importing remote assets to DPH Catalog fails:

• Display the error message AS IT IS to the user to take corrective steps. Point out the assets that failed the import with the error message, and assets that passed the import step.
• Wait for the user input to resolve the issue before proceeding with the data product creation.
• Give an option to user to proceed with the assets that are successfully processed and imported, but always wait for the user the user input.
• If user wants to proceed with the successfully processed assets, then proceed with the data product creation - call create_update_data_product_from_asset_in_container with target asset IDs of the successfully processed/imported assets.

Rollback Strategy

If publication fails partway through:

1. Draft created but not published — Draft remains in DPH, user can:

   • Fix issues and retry publication
   • Delete draft manually in DPH UI
   • Use specification files to recreate from scratch

2. Specification files preserved — User can:

   • Review what went wrong
   • Fix specification files
   • Retry entire publication process

---

Advanced Scenarios

Scenario 1: Multiple Assets from Different Containers

// assets_manifest.json
{
  "source_type": "asset",
  "assets": [
    {
      "asset_id": "asset-1",
      "container_id": "catalog-abc",
      "container_type": "catalog",
      ...
    },
    {
      "asset_id": "asset-2",
      "container_id": "project-xyz",
      "container_type": "project",
      ...
    }
  ]
}

Publication: Call import_remote_assets_to_data_product_catalog tool for all assets at once. Then call create_update_data_product_from_asset_in_container once for all target_asset_ids returned by the previous tool call.

Scenario 2: Mixed Asset and URL in one Data Product

User can choose both asset-based AND URL-based in one product, if desired.

Scenario 3: Update Existing Data Product

This skill creates new data products. Once the data product is published, it is immutable. Therefore, to update an existing data product, the option is to:

• Create a new version using this skill with updated specifications

Scenario 4: Template for Common Patterns

After creating a data product, suggest:

Tip: You can use these specification files as a template!
 
To create a similar data product:
1. Copy .data_products/customer-360/ to .data_products/new-product/
2. Edit the JSON files with new values
3. Run this skill again
 
This is especially useful for:
- Creating data products for different regions
- Onboarding similar datasets
- Standardizing data product structure across teams

---

Key Principles

1. Files First, API Calls Last: Generate complete specifications before touching DPH
2. Validate Early: Catch errors in files before making API calls
3. Iterate Freely: User can refine specifications without API rate limits or side effects
4. Preserve History: Files can be versioned in Git
5. Batch Operations: Make 5-7 API calls instead of 15-20
6. Clear Communication: Always explain what you're doing and why
7. User Confirmation: Never publish without explicit user approval
8. Spec Files Are Always Source of Truth: If there is any change request, update the specification file first before proceeding.

---

Critical Reminders

1. Treat specification files as source of truth - DPH is the deployment target
2. Never publish without user confirmation - always validate first
3. Preserve specification files - they enable version control and reusability
4. Validate before API calls - catch errors early in the workflow
5. Batch operations - minimize API calls by using complete specifications

---