// Reference for merge-train automation internals -- workflows, scripts, CI integration, and configuration. Use when modifying or debugging merge-train infrastructure.
Run realistic Chonk (client IVC) benchmarks using pinned protocol inputs. Covers native and WASM proving, per-circuit breakdowns, BB_BENCH instrumentation, and profiling code augmentation. Use when asked to benchmark, profile, or measure Chonk proving performance.
Run the Chonk (client-IVC) prover on the remote EC2 and collect Perfetto-compatible JSON traces. Supports both native and WASM runtimes. Generates a one-click Perfetto UI link for visual analysis. Use when asked to profile, trace, or visualize Chonk proving performance.
Run benchmarks on the dedicated remote EC2 benchmarking machine for noise-free, single-run results. Handles env var validation, lock management, binary transfer, and result collection. Use with /benchmark-chonk or any BB benchmark target.
Analyze source-file changes referenced by documentation, update affected docs when needed, and report documentation reference updates for a PR.
Build and update the developer documentation site for a new release
Manage pinned Chonk IVC inputs and the Chonk/rollup UltraHonk proving checks. Use when updating, testing, benchmarking, or reviewing the CI flow for Chonk input refreshes.
| name | merge-train-infra |
| description | Reference for merge-train automation internals -- workflows, scripts, CI integration, and configuration. Use when modifying or debugging merge-train infrastructure. |
This skill covers the automation internals of the merge-train system. For contributor-facing guidance (creating PRs, labels, handling failures), see the merge-trains skill.
The merge-train system is fully automated via GitHub Actions in .github/workflows/merge-train-*.yml:
PR Creation (merge-train-create-pr.yml): Triggered on push to merge-train/* branches. Creates a PR targeting next with the ci-no-squash label (and ci-full-no-test-cache for spartan). Skips merge commits and commits already in next.
Body Updates (merge-train-update-pr-body.yml): Triggered on push to merge-train/** and backport-to-*-staging branches. Updates the PR body with meaningful commits (those containing PR references like (#1234)). The body wraps the commit list in BEGIN_COMMIT_OVERRIDE / END_COMMIT_OVERRIDE markers. Backport staging PRs also call update-pr-body.sh inline from scripts/backport_to_staging.sh to handle the first-push case (where the PR doesn't exist yet when the workflow fires).
Next Integration (merge-train-next-to-branches.yml): Triggered on push to next. Merges next into each active merge-train branch via scripts/merge-train/merge-next.sh. Uses continue-on-error: true so a conflict in one branch does not block others. Skips branches whose PR already has auto-merge enabled.
Auto-Merge (merge-train-auto-merge.yml): Runs hourly via cron (0 * * * *). Calls scripts/merge-train/auto-merge.sh for both merge-train (4-hour inactivity) and backport-train (8-hour inactivity) branches. Uses separate GitHub tokens: AZTEC_BOT_GITHUB_TOKEN for API calls and MERGE_TRAIN_GITHUB_TOKEN for approvals. Will not auto-merge if the last merge-queue CI run failed or was cancelled.
Recreation & Wakeup (merge-train-recreate.yml): Triggered when a PR is closed (merged). If the merged PR's head branch starts with merge-train/, recreates the branch from the base branch (usually next). Then runs scripts/merge-train/wakeup-prs.sh to add the ci-wakeup-pr-after-merge label to all open PRs targeting the branch that have passed CI and have automerge enabled. This triggers a CI re-run (typically a no-op via tree-hash cache) so those PRs can proceed through the merge queue. The label is immediately removed by a step in ci3.yml so it can be re-applied on subsequent merges.
Failure Notification (merge-queue-dequeue-notify.yml): Triggered when a PR is dequeued from the merge queue. If the PR's head branch starts with merge-train/ and the PR was NOT merged, sends a Slack notification via ci3/merge_train_failure_slack_notify.
.github/ci3_labels_to_env.sh)Merge-train branches influence CI mode:
merge_group events or ci-merge-queue label → merge-queue modemerge-train/spartan → upgraded to merge-queue-heavy mode (10 parallel grind runs instead of 4)merge-train/docs → ci-docs modemerge-train/barretenberg → ci-barretenberg mode.github/workflows/ci3.yml)group: ci3-${{ (startsWith(github.event.pull_request.head.ref, 'merge-train/') && github.run_id) || ... }}
Merge-train PRs get full concurrency (each run has its own unique group via github.run_id), while non-merge-train PRs share a group by branch name with cancel-in-progress.
.github/ci3.sh)if [[ "${PR_HEAD_REF:-}" == merge-train/* ]]; then
export INSTANCE_POSTFIX=${PR_COMMITS:-}
fi
Merge-train PRs get a unique instance postfix (commit count) to allow parallel EC2 instances.
ci-docs: Only builds and tests documentationci-barretenberg: Only builds and tests barretenberg (AVM disabled)ci-barretenberg-full: Full barretenberg CI including acir_testsmerge-queue: 4x AMD64 full + 1x ARM64 fast in parallelmerge-queue-heavy: 10x AMD64 full + 1x ARM64 fast in parallel (used for merge-train/spartan)ci3/run_test_cmd)if [[ "$is_merge_queue" -eq 1 || ("${TARGET_BRANCH:-}" =~ ^v[0-9]) || ("${TARGET_BRANCH:-}" == merge-train/*) ]]; then
track_test_history=1
fi
ci3/bootstrap_ec2)When a CI run fails on an EC2 instance, it calls merge_train_failure_slack_notify to send failure notifications to the appropriate Slack channel based on the branch name.
next with naming pattern merge-train/{team}.github/workflows/merge-train-next-to-branches.ymlci3/merge_train_failure_slack_notify.github/ci3_labels_to_env.sh and bootstrap.sh| File | Purpose |
|---|---|
.github/workflows/merge-train-readme.md | User-facing documentation |
.github/workflows/merge-train-create-pr.yml | Auto-creates PRs for train branches |
.github/workflows/merge-train-auto-merge.yml | Hourly cron to auto-merge inactive trains |
.github/workflows/merge-train-next-to-branches.yml | Syncs next into all train branches; defines active branches |
.github/workflows/merge-train-recreate.yml | Recreates branch after merge |
.github/workflows/merge-train-update-pr-body.yml | Updates PR body with commit list (merge-train and backport branches) |
.github/workflows/merge-queue-dequeue-notify.yml | Slack notification on merge-queue dequeue |
.github/workflows/squashed-pr-check.yml | Squash enforcement (skipped for ci-no-squash) |
| File | Purpose |
|---|---|
scripts/merge-train/auto-merge.sh | Auto-merge logic -- checks inactivity, last CI status, approves and merges |
scripts/merge-train/merge-next.sh | Merges next into a train branch, handles conflicts, cancels stale CI runs |
scripts/merge-train/update-pr-body.sh | Updates PR body with meaningful commits |
scripts/merge-train/squash-pr.sh | Squashes PR commits (used by ci-squash-and-merge label) |
scripts/merge-train/wakeup-prs.sh | Adds ci-wakeup-pr-after-merge label to qualifying PRs after branch recreation |
scripts/backport_to_staging.sh | Cherry-picks a merged PR to a backport staging branch; creates/updates the backport PR |
| File | Purpose |
|---|---|
.github/ci3_labels_to_env.sh | CI mode selection based on labels and target branches |
.github/ci3.sh | Instance postfix for merge-train parallelism |
ci3/merge_train_failure_slack_notify | Slack failure notification with branch-to-channel mapping |
ci3/run_test_cmd | Test history tracking for merge-train branches |
ci3/bootstrap_ec2 | EC2 failure notification trigger |
bootstrap.sh | CI mode definitions (ci-docs, ci-barretenberg, etc.) |
| File | Purpose |
|---|---|
scripts/auto_close_issues.py | Auto-closes issues referenced in merged merge-train PRs (GitHub's native auto-close doesn't work for intermediate branches) |
scripts/find_orphaned_issues_in_prs.py | Finds PRs in merge-train commits that reference still-open issues |
scripts/dedupe_release_notes.py | Deduplicates release notes from merge-train merges |
scripts/commits | Pretty git log that groups merge-train children by subsystem |
scripts/filter_history | Filters git history, identifying merge-train merge commits as "containers" |