buildkite-agent-runtime

# Buildkite Agent Runtime The `buildkite-agent` binary provides subcommands for interacting with Buildkite from within running job steps — creating annotations, uploading artifacts, sharing state between jobs, generating dynamic pipelines, requesting OIDC tokens, and more. This skill covers the command syntax, flags, and patterns for every in-job subcommand. ## Quick Start A step that runs tests, annotates failures, uploads coverage, and stores a result flag for downstream jobs: ```yaml steps: - label: ":test_tube: Tests" command: | if ! make test 2>&1 | tee test-output.txt; then buildkite-agent annotate --style "error" --context "test-failures" < test-output.txt buildkite-agent meta-data set "tests-passed" "false" exit 1 fi buildkite-agent annotate "All tests passed :white_check_mark:" --style "success" --context "test-results" buildkite-agent artifact upload "coverage/**/*" buildkite-agent meta-data set "tests-passed" "true" ``` A downstream step reading that state: ```yaml - label: ":rocket: Deploy" command: | PASSED=$(buildkite-agent meta-data get "tests-passed") if [[ "$PASSED" != "true" ]]; then echo "Tests did not pass, skipping deploy" exit 0 fi scripts/deploy.sh depends_on: "test-step" ``` ## Annotations Surface build results directly on the build page. Annotations support Markdown and HTML. ### Creating annotations ```bash # Simple text annotation buildkite-agent annotate "Deploy completed successfully" --style "success" --context "deploy" # Pipe from a file buildkite-agent annotate --style "error" --context "test-failures" < test-output.md ``` ### Key flags | Flag | Short | Default | Description | |------|-------|---------|-------------| | `--style` | `-s` | `default` | Visual style: `default`, `info`, `warning`, `error`, `success` | | `--context` | `-c` | random UUID | Unique ID — reusing a context replaces the annotation | | `--append` | — | `false` | Append to existing annotation with same context instead of replacing | | `--priority` | — | `3` | Display priority (1-10). Higher numbers appear first | | `--job` | — | current job | Job ID to annotate (rarely needed) | ### Replacing vs appending - **Same `--context` without `--append`** replaces the annotation; **with `--append`** appends below existing content. - Always use a stable `--context` value so reruns update the same annotation instead of creating duplicates. > For pipeline-level `notify:` configuration, see the **buildkite-pipelines** skill. ## Artifacts Upload files as build artifacts, download them in later steps or other builds, and search by glob. ### Upload ```bash # Upload a single file buildkite-agent artifact upload "pkg/release.tar.gz" # Upload with glob pattern buildkite-agent artifact upload "dist/**/*" ``` ### Download ```bash # Download to current directory buildkite-agent artifact download "pkg/release.tar.gz" . # Download from a specific step buildkite-agent artifact download "dist/*" . --step "build-step" ``` ### Search ```bash # List matching artifacts buildkite-agent artifact search "pkg/*.tar.gz" --build "$BUILDKITE_BUILD_ID" ``` For complete flag tables, see `references/flag-reference.md`. > For the declarative `artifact_paths:` YAML key, see the **buildkite-pipelines** skill. For `bk artifact` CLI commands, see the **buildkite-cli** skill. ## Meta-data A build-wide key-value store for sharing state between jobs. Set a value in one job, read it in any other job in the same build. ### Set ```bash buildkite-agent meta-data set "release-version" "1.4.2" ``` ### Get ```bash VERSION=$(buildkite-agent meta-data get "release-version") ``` Use `--default` to return a fallback value instead of a non-zero exit when the key is missing: `buildkite-agent meta-data get "deploy-env" --default "staging"`. ### Check existence ```bash # Returns exit code 0 if exists, 100 if not if buildkite-agent meta-data exists "release-version"; then echo "Version already set" fi ``` ### Common patterns **Block step field values** are stored automatically as meta-data. Retrieve them by field key: ```bash # After a block step with fields: [{key: "release-name", text: "Release Name"}] RELEASE_NAME=$(buildkite-agent meta-data get "release-name") ``` **Raw webhook payloads** are available in webhook-triggered builds, while the webhook data remains cached. Read the special `buildkite:webhook` key when automation needs fields that are not promoted to first-class environment variables, such as the original pull request comment body. ```bash WEBHOOK="$(buildkite-agent meta-data get "buildkite:webhook")" COMMENT_BODY="$(jq -r '.comment.body' <<< "$WEBHOOK")" ``` ## Pipeline Upload Dynamically add steps to a running build. The core mechanism behind dynamic pipelines — generate YAML at runtime and upload it. ### Basic usage ```bash # Upload a specific file buildkite-agent pipeline upload .buildkite/deploy-steps.yml # Pipe generated YAML from stdin ./scripts/generate-pipeline.sh | buildkite-agent pipeline upload ``` ### Replace mode By default, uploaded steps are **appended** after the current step. Use `--replace` to replace the entire remaining pipeline: ```bash # Replace all remaining steps with the uploaded ones buildkite-agent pipeline upload --replace .buildkite/new-pipeline.yml ``` ### Key flags | Flag | Default | Description | |------|---------|-------------| | `--replace` | `false` | Replace remaining pipeline steps instead of appending | | `--no-interpolation` | `false` | Skip environment variable interpolation in the uploaded YAML | | `--dry-run` | `false` | Validate and output the pipeline without uploading | > For pipeline YAML syntax and step types, see the **buildkite-pipelines** skill. ## OIDC Tokens Request short-lived OpenID Connect tokens from within a job step for authenticating to external services (cloud providers, package registries) without static credentials. ### Basic token request ```bash # Request a token for a specific audience TOKEN=$(buildkite-agent oidc request-token --audience "https://packages.buildkite.com/my-org/my-registry") ``` ### Cloud provider authentication ```bash # AWS — request token with STS audience TOKEN=$(buildkite-agent oidc request-token --audience "sts.amazonaws.com") ``` ### Key flags | Flag | Default | Description | |------|---------|-------------| | `--audience` | Buildkite endpoint | Target service URL — must match the OIDC provider audience configuration | | `--lifetime` | `0` (API default) | Token lifetime in seconds. When 0 or omitted, the API chooses a default lifetime | | `--claim` | — | Optional claims to include (e.g., `organization_id,pipeline_id`). Repeatable | | `--aws-session-tag` | — | Claims to map as AWS session tags. Repeatable | > For end-to-end OIDC auth flows, cloud provider setup, and token claim details, see the **buildkite-secure-delivery** skill. ## Step Management Read or modify step attributes at runtime. Useful for conditional logic within steps and build automation. ### Get step attributes ```bash # Get a step's label (--step is required) LABEL=$(buildkite-agent step get "label" --step "$BUILDKITE_STEP_KEY") # Get another step's state STATE=$(buildkite-agent step get "state" --step "deploy-step") # Get the entire step as JSON buildkite-agent step get --step "test-step" --format json ``` ### Update step attributes ```bash # Update a step's label dynamically buildkite-agent step update "label" ":rocket: Deploying v${VERSION}" --step "$BUILDKITE_STEP_KEY" # Append to an existing label buildkite-agent step update "label" " (retried)" --step "build-step" --append ``` ### Cancel a step ```bash # Cancel all unfinished jobs for a step buildkite-agent step cancel --step "optional-step" # Force-cancel a running step buildkite-agent step cancel --step "stuck-step" --force ``` ### Key flags | Flag | Default | Description | |------|---------|-------------| | `--step` | — | Step key or UUID to target (**required** — uses `$BUILDKITE_STEP_ID` env var if set) | | `--build` | current build | Build UUID (for cross-build operations) | | `--format` | — | Output format for `get` (use `json` for complex attributes) | | `--append` | `false` | Append to existing value instead of replacing (`step update` only) | | `--force` | `false` | Force cancel even if the step is running (`step cancel` only) | ## Distributed Locks Coordinate parallel jobs within a build using distributed mutex locks. Prevents race conditions when multiple jobs access shared resources. ### Acquire / release pattern ```bash #!/bin/bash set -euo pipefail # Acquire lock — blocks until available, returns a token token=$(buildkite-agent lock acquire "database-migration") trap 'buildkite-agent lock release "database-migration" "${token}"' EXIT # Critical section — only one job runs this at a time bundle exec rails db:migrate ``` ### Do / done pattern (one-time setup) Run a setup task exactly once across all parallel jobs: ```bash #!/bin/bash echo "+++ Setting up shared test environment" if [[ $(buildkite-agent lock do "test-env-setup") == "do" ]]; then echo "Downloading test assets..." curl -o /tmp/test-data.zip https://releases.example.com/data.zip unzip /tmp/test-data.zip -d /tmp/shared-test-files/ buildkite-agent lock done "test-env-setup" else echo "Assets already prepared by another job" fi # All jobs continue here run-tests.sh ``` ### Key flags | Subcommand | Flags | Description | |------------|-------|-------------| | `lock acquire <name>` | `--lock-wait-timeout` | Maximum wait duration (e.g. `30s`, `5m`). Default: wait forever | | `lock release <name> <token>` | — | Release with the token from `acquire` | | `lock do <name>` | — | Returns `do` if lock acquired, `done` if already completed | | `lock done <name>` | — | Mark a `do` lock as completed | ## Environment Inspect and modify the job's environment variables. Primarily useful for debugging lifecycle hooks and understanding what environment changes hooks made. ```bash # Dump all environment variables as JSON buildkite-agent env dump | jq . # Get a specific variable buildkite-agent env get "BUILDKITE_BRANCH" # Set variables for subsequent hooks and the command (KEY=value format) buildkite-agent env set DEPLOY_TARGET=production "APP_NAME=My App" ``` ### Key flags | Subcommand | Description | |------------|-------------| | `env dump` | Dump all environment variables (JSON format by default) | | `env get <keys...>` | Get one or more specific variables. Use `--format` for output format (`plain`, `json`, `json-pretty`) | | `env set KEY=value [...]` | Set variables for subsequent phases. Accepts multiple `KEY=value` pairs | | `env unset <key>` | Remove a variable from subsequent phases | ### Debugging hooks The `env dump` command is particularly useful in lifecycle hooks to see what prior hooks changed: ```bash #!/bin/bash # .buildkite/hooks/pre-command echo "--- Environment after environment hook:" buildkite-agent env dump | jq 'keys' ``` > For agent lifecycle hooks and `buildkite-agent.cfg` configuration, see the [agent hooks](https://buildkite.com/docs/agent/hooks) and [agent configuration](https://buildkite.com/docs/agent/self-hosted/configure) documentation. ## Secrets Retrieve cluster secrets at runtime from within job steps. Secrets retrieved this way are automatically added to the log redactor. ### Basic usage ```bash # Get a secret value SECRET_VAR=$(buildkite-agent secret get "deploy-key") # Pass directly to a tool cli-tool --token "$(buildkite-agent secret get "api-token")" ``` ### Key flags | Flag | Default | Description | |------|---------|-------------|