local-testing

# Local Testing Skill This skill documents how to properly set up a local development environment for testing, including running unit tests and CLI E2E tests with mock Claude. ## Quick Start ### Unit Tests Only ```bash # 1. Install dependencies cd turbo && pnpm install # 2. Run unit tests cd turbo && pnpm vitest run ``` ### CLI E2E Tests ```bash # 1. Start dev server with tunnel (required for webhooks) /dev-server:start # 2. Wait for server to be ready, then authenticate CLI /dev-server:auth # 3. Deploy runner (needed for vm0 run, takes several minutes) /dev-server:runner # 4. Run CLI E2E tests VM0_API_URL=http://localhost:3000 USE_MOCK_CLAUDE=true BATS_TEST_TIMEOUT=60 \ ./e2e/test/libs/bats/bin/bats -T ./e2e/tests/01-serial/*.bats ``` --- ## Prerequisites ### 1. Environment Variables Ensure the following are set in `turbo/apps/web/.env.local`: | Variable | Purpose | Required | |----------|---------|----------| | `USE_MOCK_CLAUDE` | Enable mock Claude for testing (set to `true`) | Yes for E2E | | `CONCURRENT_RUN_LIMIT_CAP` | Set to `0` to disable run limits during testing | Yes for E2E | | `SECRETS_ENCRYPTION_KEY` | Encryption key for secrets | Yes | | `CLERK_SECRET_KEY` | Clerk authentication | Yes | | `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` | Clerk authentication | Yes | **If environment variables are missing**, ask the user to run the sync script: ```bash scripts/sync-env.sh ``` > **Note**: `sync-env.sh` requires 1Password authentication and can only be executed by the user directly. If you encounter missing environment variable errors, request the user to run this script. ### 2. Add Mock Claude Configuration ```bash # Add to turbo/apps/web/.env.local echo "USE_MOCK_CLAUDE=true" >> turbo/apps/web/.env.local echo "CONCURRENT_RUN_LIMIT_CAP=0" >> turbo/apps/web/.env.local ``` --- ## Starting the Dev Server ### Using the Skill ```bash /dev-server:start ``` This will: - Start the Turbo dev server in background - Automatically start a Cloudflare tunnel for the web app - Set `VM0_API_URL` to the tunnel URL - Enable webhooks to reach your local server ### Manual Start ```bash cd turbo && pnpm dev ``` ### Verify Server is Ready Check for these indicators in the logs (`/dev-server:logs`): - `[tunnel] Tunnel URL: https://xxx.trycloudflare.com` - `Ready in XXXms` for each app - No fatal errors **Important**: The tunnel URL changes each time you restart. Sandbox webhooks use this URL to send events back to your local server. --- ## Running Unit Tests ### Prerequisites Before running unit tests, ensure dependencies are installed: ```bash cd turbo && pnpm install ``` ### Run All Tests ```bash cd turbo && pnpm vitest run ``` Expected output: ``` Test Files 173 passed (173) Tests 2348 passed (2348) ``` ### Run Tests in Watch Mode ```bash cd turbo && pnpm vitest ``` ### Run Specific Test File ```bash cd turbo && pnpm vitest run apps/web/src/lib/__tests__/my-test.test.ts ``` ### Run Tests for a Specific Package ```bash cd turbo && pnpm vitest run --project @vm0/cli cd turbo && pnpm vitest run --project web ``` --- ## Running CLI E2E Tests ### Test Structure ``` e2e/tests/ ├── 01-serial/ # Tests that MUST run serially (scope setup) └── 03-runner/ # All parallel tests (runs on runner) ``` ### Environment Variables for E2E | Variable | Value | Purpose | |----------|-------|---------| | `VM0_API_URL` | `http://localhost:3000` | API endpoint | | `USE_MOCK_CLAUDE` | `true` | Use mock Claude instead of real API | | `BATS_TEST_TIMEOUT` | `30` (serial) / `60` (parallel) | Per-test timeout in seconds | ### Running Serial Tests ```bash VM0_API_URL=http://localhost:3000 \ USE_MOCK_CLAUDE=true \ BATS_TEST_TIMEOUT=30 \ ./e2e/test/libs/bats/bin/bats -T ./e2e/tests/01-serial/*.bats ``` ### Running Parallel Tests ```bash VM0_API_URL=http://localhost:3000 \ USE_MOCK_CLAUDE=true \ BATS_TEST_TIMEOUT=60 \ ./e2e/test/libs/bats/bin/bats -T -j 10 --no-parallelize-within-files ./e2e/tests/03-runner/*.bats ``` ### Running a Single Test File ```bash VM0_API_URL=http://localhost:3000 \ USE_MOCK_CLAUDE=true \ BATS_TEST_TIMEOUT=60 \ ./e2e/test/libs/bats/bin/bats -T ./e2e/tests/03-runner/t17-vm0-simplified-compose.bats ``` ### Running a Specific Test by Name ```bash VM0_API_URL=http://localhost:3000 \ USE_MOCK_CLAUDE=true \ BATS_TEST_TIMEOUT=60 \ ./e2e/test/libs/bats/bin/bats -T ./e2e/tests/03-runner/t17-vm0-simplified-compose.bats \ --filter "vm0 compose with both instructions and skills" ``` --- ## How Mock Claude Works When `USE_MOCK_CLAUDE=true`: 1. The web server passes this to the sandbox via environment variable 2. Inside sandbox, `run-agent.ts` checks for `USE_MOCK_CLAUDE` 3. Instead of running real `claude` CLI, it runs the mock Claude script 4. Mock Claude executes the prompt as a bash command and outputs Claude-compatible JSONL --- ## Troubleshooting ### Unit Test Issues #### Problem: "@radix-ui/react-select could not be resolved" or Missing Dependencies **Cause**: Dependencies not installed or out of sync **Solution**: ```bash cd turbo && pnpm install ``` #### Problem: Tests Fail After Pulling New Changes **Solution**: ```bash cd turbo && pnpm install cd turbo && pnpm vitest run ``` --- ### CLI E2E Test Issues ### Problem: "Failed to fetch events" or Run Hangs **Cause**: Sandbox cannot reach the tunnel URL (webhooks fail) **Solution**: 1. Check if tunnel is active: `/dev-server:logs tunnel` 2. Test tunnel connectivity: ```bash curl -v https://<tunnel-url>/api/webhooks/agent/events ``` 3. If SSL errors occur, restart dev server to get a fresh tunnel: ```bash /dev-server:stop /dev-server:start ``` ### Problem: "Concurrent agent run limit" Error **Cause**: `CONCURRENT_RUN_LIMIT_CAP` not set or not 0 **Solution**: ```bash echo "CONCURRENT_RUN_LIMIT_CAP=0" >> turbo/apps/web/.env.local # Then restart dev server ``` ### Problem: Tests Timeout **Cause**: `vm0 run` takes ~15-30 seconds per execution **Solution**: - Ensure `BATS_TEST_TIMEOUT` is set appropriately (60s for parallel tests) - Check server logs for errors: `/dev-server:logs error` - Verify sandbox is starting: `/dev-server:logs sandbox` ### Problem: "SECRETS_ENCRYPTION_KEY" Missing or Other Environment Variables Missing **Cause**: Environment variables not synced from 1Password **Solution**: Ask the user to run the sync script (requires 1Password authentication): ```bash scripts/sync-env.sh ``` > **Note**: This script can only be executed by the user directly as it requires interactive 1Password authentication. ### Problem: SSL Certificate Errors **Cause**: `CF_DNS_AND_TUNNEL_API_TOKEN` not set. Caddy needs this to provision Let's Encrypt certificates via DNS-01 challenge. **Solution**: ```bash scripts/sync-env.sh ``` ### Problem: "parallel: command not found" When Running Parallel E2E Tests **Cause**: GNU Parallel not installed **Solution**: ```bash sudo apt-get install -y parallel ``` ### Problem: Port Already in Use **Solution**: ```bash # Kill processes on dev ports fuser -k 3000/tcp 3001/tcp 3002/tcp 3003/tcp # Or use dev-stop /dev-server:stop ``` ### Problem: "SyntaxError: Unexpected end of JSON input" During Parallel Tests **Cause**: High concurrency causing request body truncation (local dev environment limitation) **Solution**: - This is a transient issue in local dev, CI environment doesn't have this problem - Run fewer parallel jobs: `-j 4` instead of `-j 10` - Or run tests individually to verify they pass --- ## CI vs Local Differences | Aspect | CI | Local | |--------|-----|-------| | Server | Vercel Preview | localhost + tunnel | | Tunnel | Not needed | Cloudflare tunnel required | | Concurrency | High | May have issues with -j 10 | | Timeouts | 8 min total | No global limit | --- ## Useful Commands ```bash # Check dev server status /dev-server:logs # Filter logs by pattern /dev-server:logs error /dev-server:logs tunnel /dev-server:logs sandbox # Stop dev server /dev-server:stop # Authenticate CLI with local server /dev-server:auth # Check CLI auth status vm0 auth status # Manual test of vm0 run with mock Claude vm0 run <agent-name> --artifact-name <artifact> "echo hello" ``` --- ## Reference - CLI E2E Testing Patterns: `.claude/skills/cli-e2e-testing/skill.md` - Dev Server Management: `.claude/skills/dev-server/skill.md` - Runner Executor: `turbo/apps/web/src/lib/run/executors/runner-executor.ts`