// Guides understanding and working with Apache Beam's CI/CD system using GitHub Actions. Use when debugging CI failures, understanding test workflows, or modifying CI configuration.
Guides YAML SDK development in Apache Beam, including environment setup, testing, and key concepts. Use when working with Beam YAML code in sdks/python/apache_beam/yaml/.
Guide on how to add and propagate new metadata fields in Apache Beam's WindowedValue, extending protos, windmill persistence, and runner interfaces to avoid metadata loss.
Guides understanding and working with Apache Beam runners (Direct, Dataflow, Flink, Spark, etc.). Use when configuring pipelines for different execution environments or debugging runner-specific issues.
Rewrite Apache Beam DoFn methods (@ProcessElement, @OnTimer, @OnWindowExpiration) to remove legacy ProcessContext or OnTimerContext usage. Use this skill when you encounter DoFn methods that use context.element(), context.output(), etc., and need to modernize them using parameter injection (@Element, @Timestamp, @Pane, OutputReceiver, MultiOutputReceiver).
Guides Python SDK development in Apache Beam, including environment setup, testing, building, and running pipelines. Use when working with Python code in sdks/python/.
Guides understanding and using the Gradle build system in Apache Beam. Use when building projects, understanding dependencies, or troubleshooting build issues.
| name | ci-cd |
| description | Guides understanding and working with Apache Beam's CI/CD system using GitHub Actions. Use when debugging CI failures, understanding test workflows, or modifying CI configuration. |
Apache Beam uses GitHub Actions for CI/CD. Workflows are located in .github/workflows/.
beam_PreCommit_*.ymlbeam_PostCommit_*.ymlnightly-master| Workflow | Description |
|---|---|
beam_PreCommit_Java.yml | Java build and tests |
beam_PreCommit_Python.yml | Python tests |
beam_PreCommit_Go.yml | Go tests |
beam_PreCommit_RAT.yml | License header checks |
beam_PreCommit_Spotless.yml | Code formatting |
| Workflow | Description |
|---|---|
beam_PostCommit_Java.yml | Full Java test suite |
beam_PostCommit_Java_ValidatesRunner_*.yml | Runner validation tests |
beam_PostCommit_Java_Examples_*.yml | Example pipeline tests |
| Workflow | Description |
|---|---|
beam_PostCommit_Python.yml | Full Python test suite |
beam_PostCommit_Python_ValidatesRunner_*.yml | Runner validation |
beam_PostCommit_Python_Examples_*.yml | Examples |
| Workflow | Description |
|---|---|
beam_LoadTests_*.yml | Load testing |
beam_PerformanceTests_*.yml | I/O performance |
Use trigger files to run specific workflows.
Most workflows support manual triggering via GitHub UI.
Workflows requiring GCP access use these secrets:
GCP_PROJECT_ID - Project ID (e.g., apache-beam-testing)GCP_REGION - Region (e.g., us-central1)GCP_TESTING_BUCKET - Temp storage bucketGCP_PYTHON_WHEELS_BUCKET - Python wheels bucketGCP_SA_EMAIL - Service account emailGCP_SA_KEY - Base64-encoded service account keyRequired IAM roles:
beam_*.ymlname: Workflow Name
on:
push:
branches: [master]
pull_request:
branches: [master]
schedule:
- cron: '0 0 * * *'
workflow_dispatch:
jobs:
build:
runs-on: [self-hosted, ...]
steps:
- uses: actions/checkout@v4
- name: Run Gradle
run: ./gradlew :task:name
Check workflow file's run commands:
./gradlew :sdks:java:core:test
./gradlew :sdks:python:test
rm -rf ~/.gradle .gradlerm -rf build| Workflow | Purpose |
|---|---|
cut_release_branch.yml | Create release branch |
build_release_candidate.yml | Build RC |
finalize_release.yml | Finalize release |
publish_github_release_notes.yml | Publish notes |