execute

name	execute
description	Execute benchmark test cases in sandboxed environments with AI agents. Spins up microsandbox containers for each test case and extracts solutions.
argument-hint	[project-directory] [--tests TC-001,TC-002] [--run runId]
disable-model-invocation	true
allowed-tools	Bash(agentic-usability *) Read Glob

Execute Test Cases

Run the executor stage of the benchmark pipeline. For each test case and target, this:

1. Creates a sandboxed VM from the target image
2. Scaffolds workspace (template → setup script → per-test setup instructions)
3. Uploads PROBLEM.md with the problem statement
4. Installs the agent CLI inside the sandbox
5. Uploads public sources (docs, packages) into /workspace/sources/
6. Runs the executor agent to solve the problem
7. Extracts solution files from /workspace/solution/
8. Saves all artifacts and destroys the sandbox

echo "Arguments: $ARGUMENTS"

Options

• --tests <ids>: Comma-separated test case IDs to run (e.g., )