| name | merge-train-infra |
| description | Reference for merge-train automation internals -- workflows, scripts, CI integration, and configuration. Use when modifying or debugging merge-train infrastructure. |
Merge-Train Infrastructure
This skill covers the automation internals of the merge-train system. For contributor-facing guidance (creating PRs, labels, handling failures), see the merge-trains skill.
Automation Lifecycle
The merge-train system is fully automated via GitHub Actions in .github/workflows/merge-train-*.yml:
-
PR Creation (merge-train-create-pr.yml): Triggered on push to merge-train/* branches. Creates a PR targeting next (or v5-next for -v5 trains such as merge-train/spartan-v5 and merge-train/fairies-v5) with the ci-no-squash label (plus private-port-next for any train that targets v5-next, and ci-full-no-test-cache for merge-train/spartan, merge-train/spartan-v5, and merge-train/ci). Skips merge commits and commits already in the base branch.
-
Body Updates (merge-train-update-pr-body.yml): Triggered on push to merge-train/** and backport-to-*-staging branches. Updates the PR body with meaningful commits (those containing PR references like (#1234)). The body wraps the commit list in BEGIN_COMMIT_OVERRIDE / END_COMMIT_OVERRIDE markers. Backport staging PRs also call update-pr-body.sh inline from scripts/backport_to_staging.sh to handle the first-push case (where the PR doesn't exist yet when the workflow fires).
-
Next Integration (merge-train-next-to-branches.yml): Triggered on push to next and v5-next. A push to next merges next into each next-based train; a push to v5-next merges v5-next into the -v5 trains (merge-train/spartan-v5, merge-train/fairies-v5). Both go through scripts/merge-train/merge-next.sh, which takes an optional second argument for the source branch (defaults to next). Uses continue-on-error: true so a conflict in one branch does not block others. Skips branches whose PR already has auto-merge enabled.
-
Auto-Merge (merge-train-auto-merge.yml): Runs hourly via cron (0 * * * *). Calls scripts/merge-train/auto-merge.sh for both merge-train (4-hour inactivity) and backport-train (8-hour inactivity) branches. Uses separate GitHub tokens: AZTEC_BOT_GITHUB_TOKEN for API calls and MERGE_TRAIN_GITHUB_TOKEN for approvals. Will not auto-merge if the last merge-queue CI run failed or was cancelled.
-
Recreation & Wakeup (merge-train-recreate.yml): Triggered when a PR is closed (merged). If the merged PR's head branch starts with merge-train/, recreates the branch from the base branch (usually next). Then runs scripts/merge-train/wakeup-prs.sh to add the ci-wakeup-pr-after-merge label to all open PRs targeting the branch that have passed CI and have automerge enabled. This triggers a CI re-run (typically a no-op via tree-hash cache) so those PRs can proceed through the merge queue. The label is immediately removed by a step in ci3.yml so it can be re-applied on subsequent merges.
-
Failure Notification (merge-queue-dequeue-notify.yml): Triggered when a PR is dequeued from the merge queue. If the PR's head branch starts with merge-train/ and the PR was NOT merged, sends a Slack notification via ci3/merge_train_failure_slack_notify. That script also kicks off a ClaudeBox session to investigate/fix the dequeued PR (ci3/slack_notify_with_claudebox_kickoff), passing --repo "$GITHUB_REPOSITORY" so the session runs in the mode matching the repo the train lives on. When the train is on a private mirror (…-private), claudebox.yml selects private mode; otherwise it stays public. Without that repo hint a private-train fix session lands in public mode and cannot read the PR or open the fix.
CI Integration Details
CI Mode Selection (.github/ci3_labels_to_env.sh)
Merge-train branches influence CI mode:
merge_group events or ci-merge-queue label → merge-queue mode
- If the merge-group event is for
merge-train/spartan-v5 → upgraded to merge-queue-heavy mode (10 parallel grind runs instead of 4)
- Target branch
merge-train/docs → ci-docs mode
- Target branch
merge-train/barretenberg → ci-barretenberg mode
CI Concurrency (.github/workflows/ci3.yml)
group: ci3-${{ (startsWith(github.event.pull_request.head.ref, 'merge-train/') && github.run_id) || ... }}
Merge-train PRs get full concurrency (each run has its own unique group via github.run_id), while non-merge-train PRs share a group by branch name with cancel-in-progress.
Instance Postfix (.github/ci3.sh)
if [[ "${PR_HEAD_REF:-}" == merge-train/* ]]; then
export INSTANCE_POSTFIX=${PR_COMMITS:-}
fi
Merge-train PRs get a unique instance postfix (commit count) to allow parallel EC2 instances.
CI Modes in bootstrap.sh
ci-docs: Only builds and tests documentation
ci-barretenberg: Only builds and tests barretenberg (AVM disabled)
ci-barretenberg-full: Full barretenberg CI including acir_tests
merge-queue: 4x AMD64 full + 1x ARM64 fast in parallel
merge-queue-heavy: 10x AMD64 full + 1x ARM64 fast in parallel (used for merge-train/spartan and merge-train/spartan-v5)
Test History Tracking (ci3/run_test_cmd)
if [[ "$is_merge_queue" -eq 1 || ("${TARGET_BRANCH:-}" =~ ^v[0-9]) || ("${TARGET_BRANCH:-}" == merge-train/*) ]]; then
track_test_history=1
fi
Failure Notification (ci3/bootstrap_ec2)
When a CI run fails on an EC2 instance, it calls merge_train_failure_slack_notify to send failure notifications to the appropriate Slack channel based on the branch name.
Creating a New Merge Train
- Create a branch from the desired base (
next for most trains; a release line like v5-next for a release-specific train) with naming pattern merge-train/{team}. For a v5-release train use the -v5 suffix (merge-train/{team}-v5): merge-train-create-pr.yml routes any *-v5 branch to a v5-next base automatically and adds the private-port-next label.
- Add the branch to the loop in
.github/workflows/merge-train-next-to-branches.yml. If it tracks a base branch other than next, also add that base to the workflow's push trigger and pass it as the second argument to merge-next.sh (see the v5-next → -v5 trains wiring)
- For a base other than
next that the *-v5 convention does not already cover, set the PR base in .github/workflows/merge-train-create-pr.yml. Either way, add a stale-check job that passes BASE_BRANCH in .github/workflows/merge-train-stale-check.yml
- Add the branch-to-Slack-channel mapping in
ci3/merge_train_failure_slack_notify
- Optionally add CI mode overrides in
.github/ci3_labels_to_env.sh and bootstrap.sh
- Push code to the branch -- automation handles PR creation from there
Key Files Reference
Workflows
| File | Purpose |
|---|
.github/workflows/merge-train-readme.md | User-facing documentation |
.github/workflows/merge-train-create-pr.yml | Auto-creates PRs for train branches |
.github/workflows/merge-train-auto-merge.yml | Hourly cron to auto-merge inactive trains |
.github/workflows/merge-train-next-to-branches.yml | Syncs next into all train branches; defines active branches |
.github/workflows/merge-train-recreate.yml | Recreates branch after merge |
.github/workflows/merge-train-update-pr-body.yml | Updates PR body with commit list (merge-train and backport branches) |
.github/workflows/merge-queue-dequeue-notify.yml | Slack notification on merge-queue dequeue |
.github/workflows/squashed-pr-check.yml | Squash enforcement (skipped for ci-no-squash) |
Scripts
| File | Purpose |
|---|
scripts/merge-train/auto-merge.sh | Auto-merge logic -- checks inactivity, last CI status, approves and merges |
scripts/merge-train/merge-next.sh | Merges next into a train branch, handles conflicts, cancels stale CI runs |
scripts/merge-train/update-pr-body.sh | Updates PR body with meaningful commits |
scripts/merge-train/squash-pr.sh | Squashes PR commits (used by ci-squash-and-merge label) |
scripts/merge-train/wakeup-prs.sh | Adds ci-wakeup-pr-after-merge label to qualifying PRs after branch recreation |
scripts/backport_to_staging.sh | Cherry-picks a merged PR to a backport staging branch; creates/updates the backport PR |
CI Configuration
| File | Purpose |
|---|
.github/ci3_labels_to_env.sh | CI mode selection based on labels and target branches |
.github/ci3.sh | Instance postfix for merge-train parallelism |
ci3/merge_train_failure_slack_notify | Slack failure notification with branch-to-channel mapping |
ci3/run_test_cmd | Test history tracking for merge-train branches |
ci3/bootstrap_ec2 | EC2 failure notification trigger |
bootstrap.sh | CI mode definitions (ci-docs, ci-barretenberg, etc.) |
Other Scripts
| File | Purpose |
|---|
scripts/auto_close_issues.py | Auto-closes issues referenced in merged merge-train PRs (GitHub's native auto-close doesn't work for intermediate branches) |
scripts/find_orphaned_issues_in_prs.py | Finds PRs in merge-train commits that reference still-open issues |
scripts/dedupe_release_notes.py | Deduplicates release notes from merge-train merges |
scripts/commits | Pretty git log that groups merge-train children by subsystem |
scripts/filter_history | Filters git history, identifying merge-train merge commits as "containers" |