yaml-development

name	yaml-development
description	Guides YAML SDK development in Apache Beam, including environment setup, testing, and key concepts. Use when working with Beam YAML code in sdks/python/apache_beam/yaml/.

YAML Development in Apache Beam

Project Structure

Key Files in sdks/python/apache_beam/yaml/

• integration_tests.py - Runs integration tests defined in YAML files or using testcontainers.
• main.py - Entry point for running YAML pipelines from the command line.
• pipeline.schema.yaml - JSON schema defining the valid structure for Beam YAML pipelines.
• standard_io.yaml - Declarations of standard IO transforms and their mappings to providers.
• standard_providers.yaml - Configuration for standard providers (e.g., Java expansion services).
• yaml_combine.py - Implementations for aggregation and combining operations.
• yaml_io.py - Mappings and logic for IO transforms (e.g., PubSub, BigQuery, Iceberg).
• yaml_join.py - Implementations for join operations.
• yaml_mapping.py - Implementations for mapping operations (e.g., MapToFields).
• yaml_provider.py - Manages providers (Python, Java cross-language) that implement transforms.
• yaml_transform.py - Core YAML expansion logic, parsing, and translation to Beam pipelines.

Environment Setup

Since Beam YAML is implemented within the Python SDK, the environment setup is identical to Python development. Refer to the python-development skill for details on using pyenv and installing in editable mode (e.g., use pip install -e sdks/python[gcp,test] from the root directory).