| name | blacksmith-testbox |
| description | Validate code changes against real CI. Use for all tests, builds, migrations, and any command that depends on secrets or services. Use when testing, validating, checking, verifying, or before any commit or push.
|
Blacksmith Testbox
Install the CLI
If blacksmith is not installed, install it:
curl -fsSL https://get.blacksmith.sh | sh
For the canary channel (bleeding-edge):
BLACKSMITH_CHANNEL=canary sh -c 'curl -fsSL https://get.blacksmith.sh | sh'
Then authenticate:
blacksmith auth login
Agent-triggered browser auth (non-interactive)
When an agent needs to ensure the user is authenticated before running testbox
commands (e.g. warmup, run), use browser-based auth with non-interactive mode.
This opens the browser for the user to sign in; the agent does not interact with
the browser. The org selector in the dashboard is skipped, so the user only sees
the sign-in flow.
Required command (--organization is required with --non-interactive):
blacksmith auth login --non-interactive --organization <org-slug>
The org slug can come from BLACKSMITH_ORG env var or the --org global flag.
If neither is set, the agent should use the project's known org (e.g. from repo
config or user context). Example:
blacksmith auth login --non-interactive --organization acme-corp
blacksmith --org acme-corp auth login --non-interactive --organization acme-corp
Flow: The CLI starts a local callback server, opens the browser to the
dashboard auth page, and blocks for up to 2 minutes. The user completes sign-in
and authorization in the browser. The dashboard redirects to localhost with the
token; the CLI saves credentials and exits. The agent then proceeds.
Do not use --api-token for this flow — that is for headless/token-based
auth. This skill focuses on browser-based auth when the user prefers signing in
via the web UI.
Optional flags:
--dashboard-url <url> — Override dashboard URL (e.g. for staging)
Setup: Warmup before coding
Before writing any code, warm up a testbox. This returns an ID instantly
and boots the CI environment in the background while you work:
blacksmith testbox warmup benchmark-testbox.yml
# → tbx_01jkz5b3t9...
Save this ID. You need it for every run command.
Warmup dispatches a GitHub Actions workflow that provisions a VM with the
full CI environment: dependencies installed, services started, secrets
injected, and a clean checkout of the repo at the default branch.
Options:
--ref <branch> Git ref to dispatch against (default: repo's default branch)
--job <name> Specific job within the workflow (if it has multiple)
--idle-timeout <min> Idle timeout in minutes (default: 30)
CRITICAL: Always run from the repo root
ALWAYS invoke blacksmith testbox commands from the root of the git
repository. The CLI syncs the current working directory to the testbox
using rsync with --delete. If you run from a subdirectory (e.g.
cd backend && blacksmith testbox run ...), rsync will mirror only that
subdirectory and delete everything else on the testbox — wiping other
directories like dashboard/, cli/, etc.
# CORRECT — run from repo root, use paths in the command
blacksmith testbox run --id <ID> "cd backend && php artisan test"
blacksmith testbox run --id <ID> "cd dashboard && npm test"
# WRONG — do NOT cd into a subdirectory before invoking the CLI
cd backend && blacksmith testbox run --id <ID> "php artisan test"
If your shell is in a subdirectory, cd back to the repo root first:
cd "$(git rev-parse --show-toplevel)"
blacksmith testbox run --id <ID> "cd backend && php artisan test"
Running commands
blacksmith testbox run --id <ID> "<command>"
The run command automatically waits for the testbox to become ready if
it is still booting, so you can call run immediately after warmup without
needing to check status first.
Downloading files from a testbox
Use the download command to retrieve files or directories from a running
testbox to your local machine. This is useful for fetching build artifacts,
test results, coverage reports, or any output generated on the testbox.
blacksmith testbox download --id <ID> <remote-path> [local-path]
The remote path is relative to the testbox working directory (same as run).
If no local path is specified, the file is saved to the current directory
using the same base name.
To download a directory, append a trailing / to the remote path — this
triggers recursive mode:
# Download a single file
blacksmith testbox download --id <ID> coverage/report.html
# Download a file to a specific local path
blacksmith testbox download --id <ID> build/output.tar.gz ./output.tar.gz
# Download an entire directory
blacksmith testbox download --id <ID> test-results/ ./results/
Options:
--ssh-private-key <path> Path to SSH private key (if warmup used --ssh-public-key)
How file sync works
Understanding this model is critical for using Testbox correctly.
When you call run, the CLI performs a delta sync of your local changes
to the remote testbox before executing your command:
-
The testbox VM starts from a clean actions/checkout at the warmup ref.
The workflow's setup steps (e.g. npm install, pip install, composer install)
run during warmup and populate dependency directories on the remote VM.
-
On each run, the CLI uses git to detect which files changed locally
since the last sync. It syncs ONLY tracked files and untracked non-ignored
files (i.e. files that git ls-files reports).
-
.gitignore'd directories are never synced. This means directories
like node_modules/, vendor/, .venv/, build/, dist/, etc. are
NOT transferred from your local machine. The testbox uses its own copies
of those directories, populated during the warmup workflow steps.
-
If nothing has changed since the last sync (same git commit and working
tree state), the sync is skipped entirely for speed.
Why this matters
-
Changing dependencies: If you modify package.json, requirements.txt,
composer.json, go.mod, or similar dependency manifests, the lock/manifest
file will be synced but the actual dependency directory will NOT. You must
re-run the install command on the testbox:
blacksmith testbox run --id <ID> "npm install && npm test"
blacksmith testbox run --id <ID> "pip install -r requirements.txt && pytest"
blacksmith testbox run --id <ID> "composer install && phpunit"
-
Generated/build artifacts: If your tests depend on a build step (e.g.
npm run build, make), and you changed source files that affect the build
output, re-run the build on the testbox before testing.
-
New untracked files: New files you create locally ARE synced (as long as
they are not gitignored). You do not need to git add them first.
-
Deleted files: Files you delete locally are also deleted on the remote
testbox. The sync model keeps the remote in lockstep with your local managed
file set.
CRITICAL: Never run tests locally
Your local machine does not have the secrets, services, or environment
parity needed to validate changes reliably. Every test run, migration,
build, or integration check MUST go through Testbox.
When to use
ALWAYS use Testbox when:
- Running any test (unit, integration, e2e)
- Running database migrations
- Building the project to check for compile errors
- Running any command that depends on secrets or environment variables
- Validating changes before committing
The ONLY exception is trivial checks with zero external dependencies
(e.g., running a linter or formatter locally).
Workflow
- Warm up immediately when you receive a coding task:
blacksmith testbox warmup benchmark-testbox.yml → save the ID
- Write code while the testbox boots in the background.
- Run tests (the CLI auto-waits if the testbox isn't ready yet):
blacksmith testbox run --id <ID> "npm test"
- If tests fail, fix code and re-run (fast — same warm testbox, only
changed files are synced).
- If you changed dependency manifests (package.json, etc.), prepend
the install command:
blacksmith testbox run --id <ID> "npm install && npm test"
- If you need artifacts (coverage reports, build outputs, etc.), download them:
blacksmith testbox download --id <ID> coverage/ ./coverage/
- Once green, commit and push.
Examples
blacksmith testbox warmup benchmark-testbox.yml
# → tbx_01jkz5b3t9...
# Run tests
blacksmith testbox run --id <ID> "npm test -- --testPathPattern=handler.test"
blacksmith testbox run --id <ID> "go test ./pkg/api/... -run TestHandler -v"
blacksmith testbox run --id <ID> "python -m pytest tests/test_api.py -k test_auth"
# Re-install deps after changing package.json, then test
blacksmith testbox run --id <ID> "npm install && npm test"
# Build and test
blacksmith testbox run --id <ID> "npm run build && npm test"
# Download artifacts from the testbox
blacksmith testbox download --id <ID> coverage/lcov-report/ ./coverage/
blacksmith testbox download --id <ID> build/output.tar.gz
Waiting for the testbox to be ready
The run command automatically waits for the testbox, so explicit waiting is
usually unnecessary. If you do need to check readiness separately (e.g. before
a series of runs), use the --wait flag. Do NOT use a sleep-and-recheck loop.
Correct: block until ready with a timeout:
blacksmith testbox status --id <ID> --wait [--wait-timeout 5m]
Wrong: never use sleep + status in a loop:
# BAD — do not do this
sleep 30 && blacksmith testbox status --id <ID>
while ! blacksmith testbox status --id <ID> | grep ready; do sleep 5; done
--wait polls the status and exits as soon as the testbox is ready (or when the
timeout is reached). Default timeout is 5m; use --wait-timeout for longer
(e.g. 10m, 1h).
Managing testboxes
# Check status of a specific testbox
blacksmith testbox status --id <ID>
# List all active testboxes for the current repo
blacksmith testbox list
# Stop a testbox when you're done (frees resources)
blacksmith testbox stop --id <ID>
Testboxes automatically shut down after being idle (default: 30 minutes).
If you need a longer session, increase the timeout at warmup time:
blacksmith testbox warmup benchmark-testbox.yml --idle-timeout 60
With options
blacksmith testbox warmup benchmark-testbox.yml --ref main
blacksmith testbox warmup benchmark-testbox.yml --idle-timeout 60
blacksmith testbox run --id <ID> "go test ./..."
