Executes real end-to-end verification against a running application after specification implementation. Detects the application type, starts the local runtime (Docker, Node, Spring Boot, etc.), runs real tests (curl for REST APIs, Playwright for web SPAs, computer-use for desktop apps), verifies acceptance criteria from the functional specification, generates a markdown report, and tears down the environment. Use when: user asks to verify a completed spec with real tests, run e2e checks after implementation, validate acceptance criteria in a live environment, or test the feature for real after task completion.
Executes real end-to-end verification against a running application after specification implementation. Detects the application type, starts the local runtime (Docker, Node, Spring Boot, etc.), runs real tests (curl for REST APIs, Playwright for web SPAs, computer-use for desktop apps), verifies acceptance criteria from the functional specification, generates a markdown report, and tears down the environment. Use when: user asks to verify a completed spec with real tests, run e2e checks after implementation, validate acceptance criteria in a live environment, or test the feature for real after task completion.
Performs real environment verification after a specification has been implemented and cleaned up. This skill bridges the gap between unit-tested code and observable runtime behavior by:
Detecting the application type from project files
Starting the local runtime (Docker Compose, dev server, Spring Boot, etc.)
Deriving tests from [IMP] acceptance criteria in the functional specification
Executing real tests (curl, Playwright, computer-use)
Mapping results to acceptance criteria
Generating a verification report
Tearing down the environment
Input: docs/specs/[id]/ (spec folder with functional specification and tasks) Output: docs/specs/[id]/e2e-report-YYYY-MM-DD-HHMMSS.md
When to Use
Use after specs.task-implementation and to confirm the feature works in reality.
specs.code-cleanup
Use when a developer says "test it for real", "verify the API actually works", "run e2e checks", or "validate acceptance criteria live".
Use to generate evidence of feature completion before closing a specification.
Do NOT use for unit testing, static analysis, or code review — this is runtime behavioral verification only.
Arguments
Argument
Required
Description
--spec
Yes
Path to the specification folder (e.g., docs/specs/001-feature/)
--task
No
Specific task ID to limit verification scope (e.g., TASK-003)
--keep-alive
No
If present, skip teardown and leave the environment running
--timeout
No
Startup and test timeout in seconds (default: 120)
--insecure
No
If present, allow curl to use -k / --insecure (TLS bypass opt-in)
Best Practices
Safety first: Never run destructive commands (rm -rf, docker system prune, sudo).
Detect, don’t assume: Use file heuristics to determine the app type; ask the user only when ambiguous.
AC-driven: Every test must trace back to an [IMP] acceptance criterion from the functional specification.
Clean up: Always teardown unless --keep-alive is passed; warn about leftover processes.
No code changes: This skill is read-only regarding source code. It may create reports but never patches logic.
Use TodoWrite: Track progress across all 8 phases.
Instructions
Phase 1: Parse Arguments and Load Context
Parse $ARGUMENTS:
--spec (required): spec folder path. Validate that the directory exists and contains at least one functional specification file (YYYY-MM-DD--*.md). If missing or invalid, abort with an error.
--task (optional): task ID filter (e.g., TASK-003). If provided, validate that tasks/<task-id>.md exists inside the spec folder.
--keep-alive (optional): boolean flag. If present, skip teardown at the end.
--timeout (optional): positive integer in seconds. Default is 120. Validate that the value is a positive integer; if not, abort with an error.
--insecure (optional): boolean flag. If present, curl commands MAY use -k / --insecure for local development with self-signed certificates. By default, TLS bypass is forbidden (REQ-NR003).
Read the functional specification and extract:
All acceptance criteria with their taxonomy tags ([IMP], [SEF], [EXT])
Only [IMP] criteria will generate runtime tests
If --task is provided, read the task file and limit scope to its provides files and related AC.
Use TodoWrite to create a todo list for all 8 phases.
Phase 1.5: Security Validation Gate
Before any command is executed, run the following security checks:
Command Whitelist Check:
The startup command derived in Phase 3 MUST match a pattern documented in references/test-execution-patterns.md.
IF the command is NOT in the whitelist → use AskUserQuestion to request explicit user confirmation before execution.
Whitelisted commands include: docker compose up -d --build, ./mvnw spring-boot:run, ./gradlew bootRun, npm run dev, npm run start:dev, npm start, cargo tauri build --debug, cargo tauri dev, npm run electron:dev, npx electron ., open *.app, and equivalent local process launchers.
Any command containing sudo, rm -rf, docker system prune, mkfs, dd, or similar destructive operations is NOT whitelisted and SHALL be rejected.
Forbidden Pattern Scan (REQ-NR001):
Scan the derived startup command and all generated test commands for:
sudo → abort with: "Forbidden: sudo is not permitted during E2E verification."
rm -rf → abort with: "Forbidden: rm -rf is not permitted during E2E verification."
docker system prune → abort with: "Forbidden: docker system prune is not permitted during E2E verification."
Any rm, drop, destroy, prune targeting databases, volumes, or local data → abort with: "Forbidden: destructive data operations are not permitted."
IF any forbidden pattern is detected → abort immediately; do NOT proceed to startup.
TLS Enforcement Check (REQ-NR003):
For any generated curl command:
IF it contains -k or --insecure AND --insecure was NOT passed → abort with: "Forbidden: curl TLS bypass (-k / --insecure) is disabled by default. Pass --insecure to opt-in."
IF --insecure was passed → log a warning: "WARNING: TLS certificate verification is disabled. Use only for local development."
Data Integrity Pre-Check (REQ-NR004):
Before executing startup commands, inspect them for patterns that would overwrite or delete existing data (e.g., rm, drop, prune, volume deletion flags).
IF the command would modify existing databases, volumes, or local data directories → abort with: "Forbidden: startup commands must not overwrite or delete existing data."
Phase 2: Detect Application Type and Discover Port
Set PROJECT_ROOT to the directory containing .git or the parent directory of --spec.
Inspect PROJECT_ROOT for configuration files using the following heuristics (execute in order):
Docker Compose is always prioritized: If a Docker Compose file exists AND any framework config also exists, classify as Docker-managed. The compose stack defines the runtime.
If multiple non-Docker configs are detected (e.g., both pom.xml and package.json without Docker Compose):
If the spec's domain clearly indicates backend vs frontend (e.g., spec title contains "API", "backend", "service"), prefer JVM or NestJS.
If the spec's domain clearly indicates frontend (e.g., spec title contains "UI", "page", "component"), prefer Web SPA.
If still ambiguous, proceed to user prompt (step 4).
If no recognizable config is found, OR if multiple non-Docker configs exist and the spec domain is ambiguous, use AskUserQuestion with exactly these options:
"REST API"
"Web SPA"
"Desktop"
"Skip"
Port Discovery: Once the application type is known, determine the target port by inspecting framework configuration files in this order:
Log the detected type and discovered port; both will be recorded in the report.
Phase 3: Start Environment and Wait for Readiness
Read references/test-execution-patterns.md (shipped with this skill) for the command mapping. Based on detection:
Data integrity pre-flight (REQ-NR004): Before executing the startup command, verify it does not contain patterns that overwrite or delete existing databases, volumes, or local data (e.g., rm, --volumes, prune, drop). If a destructive pattern is detected, abort immediately with: "Startup aborted: command would destroy existing data."
Initialize runtime state:
STARTUP_COMMAND="" — the exact command used to start the environment
HEALTH_CHECK_METHOD="" — description of how readiness was determined
STARTUP_LOGS_FILE="$(mktemp)" — temp file capturing stdout/stderr from startup
STARTUP_TIMEOUT="${TIMEOUT:-120}" — seconds to wait for readiness
START_TIME="$(date +%s)"
STARTUP_PID="" — background process PID (for local processes)
Pre-startup port check (all types):
if lsof -i :"$TARGET_PORT" >/dev/null 2>&1 || nc -z localhost "$TARGET_PORT" 2>/dev/null; thenecho"Port $TARGET_PORT is already in use."
EXISTING_PID=$(lsof -ti:"$TARGET_PORT" | head -n1)
EXISTING_CMD=$(ps -p "$EXISTING_PID" -o comm= 2>/dev/null || echo"unknown")
if [ "$APP_TYPE" = "Docker-managed" ] && docker ps --format '{{.Names}}' 2>/dev/null | grep -q "$EXISTING_CMD"; thenecho"Existing Docker container detected on port $TARGET_PORT; reusing it."
STARTUP_COMMAND="(existing container reused)"elif [ "$APP_TYPE" = "JVM-based service" ] && echo"$EXISTING_CMD" | grep -q "java"; thenecho"Existing Java process detected on port $TARGET_PORT; reusing it."
STARTUP_COMMAND="(existing Java process reused)"elseecho"Port $TARGET_PORT is occupied by an unrelated process ($EXISTING_CMD). Free the port and retry."exit 1
fifi
Docker-managed startup:
Verify Docker daemon is reachable:
if ! docker info >/dev/null 2>&1; thenecho"Docker is not available. Please start Docker Desktop or use direct framework startup."exit 1
fi
Set STARTUP_COMMAND="docker compose up -d --build"
Execute:
cd"$PROJECT_ROOT" && docker compose up -d --build >> "$STARTUP_LOGS_FILE" 2>&1
Health check loop (timeout enforced):
HEALTH_CHECK_METHOD="docker compose ps --format json"
READY=falsewhile [ $(( $(date +%s) - START_TIME )) -lt "$STARTUP_TIMEOUT" ]; do# Option A: Docker native health status (supports both JSON array and NDJSON)if docker compose ps --format json 2>/dev/null | jq -s -e '.[] | select(.Health=="healthy")' >/dev/null 2>&1; then
READY=true
HEALTH_CHECK_METHOD="docker compose ps (HEALTHCHECK=healthy)"breakfi# Option B: Fallback port polling if no HEALTHCHECK definedif curl -sf "http://localhost:${TARGET_PORT}" >/dev/null 2>&1; then
READY=true
HEALTH_CHECK_METHOD="port polling via curl on localhost:${TARGET_PORT} (every 3s)"breakfisleep 3
[ != ];
docker compose logs --=50 >> 2>&1
docker compose down >/dev/null 2>&1 ||
1
Spring Boot startup:
Determine build tool and launch in background with log capture:
Build step (triggered when no pre-built debug binary exists or when source is newer than target):
Tauri:
if [ "$DESKTOP_FRAMEWORK" = "tauri" ]; thenif ! command -v cargo >/dev/null 2>&1; thenecho"ERROR: Rust/Cargo is not installed. Tauri build requires cargo."exit 1
fi
BUILD_COMMAND="cargo tauri build --debug"cd"$PROJECT_ROOT" && $BUILD_COMMAND >> "$STARTUP_LOGS_FILE" 2>&1
BUILD_EXIT_CODE=$?
[ -ne 0 ];
1
STARTUP_COMMAND=
Post-startup bookkeeping (all types):
Record STARTUP_COMMAND and HEALTH_CHECK_METHOD in report metadata (AC-010).
Append STARTUP_LOGS_FILE contents to the report under Raw Output (REQ-020).
If startup fails for any reason, capture all available logs and abort with a clear error message.
Phase 4: Generate and Execute Tests
CRITICAL: Only test [IMP] acceptance criteria. Translate each into one or more concrete runtime actions.
REST API Tests (curl)
Prerequisite check: Before generating any tests, verify curl is installed:
if ! command -v curl >/dev/null 2>&1; thenecho"ERROR: curl is not installed."echo"Install instructions:"echo" macOS: brew install curl"echo" Ubuntu: sudo apt-get install curl"echo" Windows: choco install curl or winget install curl"exit 1
fiif ! command -v jq >/dev/null 2>&1; thenecho"WARNING: jq is not installed. JSON body assertions will fall back to grep (less precise)."echo"Install instructions:"echo" macOS: brew install jq"echo" Ubuntu: sudo apt-get install jq"echo" Windows: choco install jq or winget install jqlang.jq"fi
1. Parse [IMP] AC for endpoint hints
For each [IMP] acceptance criterion in the specification:
Extract the HTTP method by searching for keywords: GET, POST, PUT, PATCH, DELETE (case-insensitive).
Extract the endpoint path by searching for patterns starting with / followed by alphanumeric segments, e.g., /api/users, /v1/health.
Extract the expected HTTP status code by searching for numeric patterns 2xx, 3xx, 4xx, 5xx or specific codes like 200, 201, 204, 400, 401, 403, 404, 500.
Extract expected Content-Type by searching for application/json, text/plain, text/html, etc.
Extract expected response body hints (field names, array presence, string values) from the AC text.
If an [IMP] AC does not contain a parseable endpoint path and method, mark it MANUAL CHECK REQUIRED and skip to the next criterion.
2. Discover authentication credentials
Before constructing curl commands, attempt to locate test credentials by scanning the following files in PROJECT_ROOT (in order):
If no credentials are found after scanning all files AND the AC text implies authentication is required (mentions "auth", "login", "token", "protected", "bearer", "API key"), use AskUserQuestion to prompt the user:
"Enter E2E_AUTH_TOKEN (or leave blank if none)"
"Enter E2E_USERNAME (or leave blank if none)"
"Enter E2E_PASSWORD (or leave blank if none)"
Security: Redact token values in the E2E report; show only the header name (e.g., Authorization: Bearer <redacted>).
3. Generate curl commands
For each parseable [IMP] AC, construct the curl command using this exact pattern:
Always include -s -w "\n%{http_code}" -o /tmp/e2e_resp.json.
-X <METHOD>: only add if the method is not GET. For GET, omit -X entirely.
-H "Content-Type: application/json": only add for POST, PUT, PATCH.
-d '<REQUEST_BODY>': only add when the AC describes a request body. If no body is described, omit -d.
<AUTH_HEADER>:
If E2E_AUTH_TOKEN is set: -H "Authorization: Bearer ${E2E_AUTH_TOKEN}"
If E2E_USERNAME and E2E_PASSWORD are set: -u "${E2E_USERNAME}:${E2E_PASSWORD}"
If no credentials found: omit auth header.
TLS enforcement (REQ-NR003): NEVER add -k or --insecure to curl commands unless the --insecure flag was explicitly passed when invoking the skill. If --insecure was passed, log a warning that TLS verification is disabled.
Store the raw response body in /tmp/e2e_resp.json and the status code on the last line of stdout.
4. Execute curl and assert (no retry)
Execute each curl command exactly once (REQ-NR008). Do NOT retry on failure.
Expected status code, Content-Type, body assertions
Actual status code, Content-Type, body snippet (first 500 chars)
Pass/fail status
Execution time (optional, for report reference)
If any assertion fails, mark the AC as FAILED immediately. Do not retry.
Web SPA Tests (Playwright)
Prerequisite check: Before generating any SPA tests, verify Playwright is installed:
PLAYWRIGHT_VERSION=$(npx playwright --version 2>/dev/null || echo"")
if [ -z "$PLAYWRIGHT_VERSION" ]; thenecho"ERROR: Playwright is not installed."echo"Install instructions:"echo" npm install -D @playwright/test"echo" npx playwright install chromium"echo""echo"Skipping all Web SPA tests. Acceptance criteria for SPA interactions will be marked MANUAL CHECK REQUIRED."for ac_id in $(get_spa_ac_ids); do
record_result "$ac_id""MANUAL CHECK REQUIRED""Playwright not installed"done# Continue to next test category; do not abort the whole verificationfiecho"Playwright version: $PLAYWRIGHT_VERSION"
If Playwright is missing, the skill MUST report the gap with the install commands above and skip SPA tests. Do NOT attempt to auto-install.
Each Playwright test MUST enforce a per-test timeout to prevent indefinite hangs (REQ-NR007). The wrapper script uses the timeout command:
TEST_TIMEOUT_SEC=30
npx playwright --version >/dev/null 2>&1 || {
echo"Playwright not installed; skipping SPA tests."
record_spa_manual_check
continue
}
# Build the inline test script
TEST_SCRIPT="$(mktemp /tmp/e2e-spa-XXXXXX.js)"cat > "$TEST_SCRIPT" << EOF
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
page.setDefaultTimeout(${TEST_TIMEOUT_SEC}000);
page.setDefaultNavigationTimeout(${TEST_TIMEOUT_SEC}000);
try {
await page.goto('${DEV_SERVER_URL}');
// --- GENERATED ACTIONS ---
// --- GENERATED ASSERTIONS ---
console.log('RESULT: PASS');
} catch (error) {
console.error('RESULT: FAIL:', error.message);
const screenshotPath = '${ARTIFACT_DIR}/screenshot-' + Date.now() + '-ac-${AC_ID}.png';
await page.screenshot({ path: screenshotPath, fullPage: true }).catch(() => {});
console.error('SCREENSHOT:', screenshotPath);
process.exitCode = 1;
} finally {
await browser.close();
}
})();
EOF# Run with timeout wrapper; if the test hangs, it is killed and marked FAILEDiftimeout --signal=TERM $((TEST_TIMEOUT_SEC + 5)) node "$TEST_SCRIPT"; then
STATUS="VERIFIED"else
EXIT_CODE=$?
if [ "$EXIT_CODE" -eq 124 ]; thenecho"FAIL: Test hung and was terminated after ${TEST_TIMEOUT_SEC}s timeout"fi
STATUS="FAILED"firm -f "$TEST_SCRIPT"
If the test process hangs beyond the timeout, the timeout command sends SIGTERM, the AC is marked FAILED, and the evidence records: "Test hung and was terminated after ${TEST_TIMEOUT_SEC}s".
7. Screenshot capture on failure
When any assertion or action fails:
Capture a full-page screenshot using page.screenshot({ path: ..., fullPage: true }).
Save to ${ARTIFACT_DIR}/screenshot-<timestamp>-ac-<AC_ID>.png.
Record the screenshot path in the test result evidence.
When a test passes, screenshots are optional and only captured if --capture-success is passed.
8. Record results
For each SPA test, record:
AC ID and text (truncated)
Playwright actions executed
Assertions performed
Pass/fail status
Screenshot path (on failure)
Execution time
Error message (on failure)
If any assertion fails, mark the AC as FAILED immediately. Do not retry.
Edge cases — error handling:
Playwright not installed: Report clear error with npm install -D @playwright/test and npx playwright install chromium suggestion. Mark all SPA ACs as MANUAL CHECK REQUIRED. Continue with other test categories.
Dev server not running: If page.goto() throws net::ERR_CONNECTION_REFUSED or similar, report: "Dev server not reachable at ${DEV_SERVER_URL}. Ensure the server is running before verification." Mark affected ACs as FAILED.
Browser launch failure: If Chromium fails to launch (e.g., missing system dependencies), report the error and suggest npx playwright install-deps chromium. Mark affected ACs as MANUAL CHECK REQUIRED.
Desktop Tests (Computer-use / MCP)
Prerequisite check: Before generating any desktop tests, verify that computer-use or MCP-based GUI automation tools are available. This skill does NOT auto-install these tools.
GUI_TOOLS_AVAILABLE=false
GUI_TOOL_NAME=""if [ -n "$CLAUDE_COMPUTER_USE_AVAILABLE" ] || command -v computer-use >/dev/null 2>&1; then
GUI_TOOLS_AVAILABLE=true
GUI_TOOL_NAME="computer-use"fiif [ -n "$MCP_GUI_SERVER_URL" ] || command -v mcp-gui >/dev/null 2>&1; then
GUI_TOOLS_AVAILABLE=true
GUI_TOOL_NAME="mcp-gui"fiif [ "$GUI_TOOLS_AVAILABLE" != "true" ]; thenecho"ERROR: Desktop testing tools are not available."echo"This skill requires computer-use or MCP GUI automation tools to verify desktop applications."echo""echo"To enable desktop verification:"echo" - Run in an environment with computer-use support, OR"echo" - Install and configure an MCP GUI automation server"echo""echo"Aborting desktop verification. No tests will be executed."exit 1
fiecho"Desktop GUI tool detected: $GUI_TOOL_NAME"
For each [IMP] acceptance criterion in the specification that relates to Desktop App behavior:
Extract window title hints: look for phrases like "window opens", "dialog appears", "modal shows", "title contains", "screen shows".
Extract UI element descriptions: look for button labels, input placeholders, menu items, checkbox labels, toggle names.
Extract workflow steps: look for action sequences like "click ... then ...", "select ... and press ...", "toggle ...".
Extract expected state changes: look for visual changes like "theme changes to dark", "status indicator turns green", "list updates".
If an [IMP] AC does not contain parseable desktop behavior hints (no window, element, or workflow descriptions), mark it MANUAL CHECK REQUIRED and skip to the next criterion.
3. Verify the application is running
If the desktop app was not started in Phase 3 or its process has exited, re-launch using the commands from Phase 3:
if [ -n "$STARTUP_PID" ] && ! kill -0 "$STARTUP_PID" 2>/dev/null; thenecho"Desktop app process is not running. Re-launching..."# Re-run the appropriate launch command from Phase 3 based on DESKTOP_FRAMEWORKfi
4. Verify window/UI elements via visual or accessibility-tree inspection
For each AC describing a window or UI element (AC-022):
Visual inspection (computer-use):
Capture a screenshot of the desktop.
Analyze the image for the expected window title, button label, or visual element.
Example: "Settings window opens" → capture screenshot, verify a window titled "Settings" is visible.
Query the accessibility tree for elements matching the expected name, role, or state.
Example: "Settings window opens" → query for a window element with name containing "Settings".
Example: "Toggle dark mode" → query for a switch/checkbox element with label containing "Dark mode".
5. Simulate user workflows through GUI automation
For each AC describing a user workflow (AC-023), translate the description into GUI automation actions:
AC Description Pattern
GUI Automation Action
"click [button label]"
Click the UI element with the matching accessible name or label
"fill [field] with [value]"
Focus the input field and type the value
"toggle [switch/checkbox]"
Click the toggle or checkbox element
"select [option] from [dropdown]"
Open the dropdown and click the option element
"navigate to [menu item]"
Click the menu item with the matching label
"type [value] into [field]"
Focus the field and type the value
6. Screenshot capture at each verification step
At EVERY step (before actions, after actions, and on assertions), capture a screenshot:
SCREENSHOT_PATH="${ARTIFACT_DIR}/screenshot-$(date +%s)-ac-${AC_ID}-step-${STEP_NUM}.png"# computer-use: capture screenshot and save to SCREENSHOT_PATH# MCP: use the GUI automation server's screenshot capability
Screenshots are saved for every step regardless of pass/fail status, satisfying AC-024 (SEF).
7. Per-test timeout enforcement
Each desktop test MUST enforce a per-test timeout to prevent indefinite hangs (REQ-NR007):
TEST_TIMEOUT_SEC=60
TEST_START_TIME=$(date +%s)
TEST_STATUS="RUNNING"while [ "$TEST_STATUS" = "RUNNING" ]; doif [ $(( $(date +%s) - TEST_START_TIME )) -gt "$TEST_TIMEOUT_SEC" ]; thenecho"FAIL: Desktop test hung and was terminated after ${TEST_TIMEOUT_SEC}s timeout"
STATUS="FAILED"
TEST_STATUS="TIMEOUT"
FINAL_SCREENSHOT="${ARTIFACT_DIR}/screenshot-$(date +%s)-ac-${AC_ID}-timeout.png"# capture screenshot to FINAL_SCREENSHOTbreakfi# Execute next workflow step; if all steps complete, set TEST_STATUS="COMPLETED"done
If a test times out:
Mark the AC as FAILED
Capture a final timeout screenshot
Record evidence: "Test hung and was terminated after ${TEST_TIMEOUT_SEC}s timeout"
Attempt to reset application state (close and reopen the window) before the next AC
8. Record results
For each desktop test, record:
AC ID and text (truncated)
GUI automation actions executed (step list)
Screenshots captured (list of paths with step descriptions)
Pass/fail status
Error message, timeout reason, or assertion mismatch (on failure)
If any assertion or action fails, mark the AC as FAILED immediately. Do not retry.
Edge cases — error handling:
No computer-use or MCP tools available: Abort verification with a clear error message. Do NOT silently skip desktop tests.
Build failure: Build output is captured in $STARTUP_LOGS_FILE and reported. Verification aborts before launch.
App crash during test: If the desktop process exits during testing, capture any crash output from logs, mark the current AC as FAILED, and attempt to restart the app for the next AC.
Window/element not found: If the expected window or element does not appear within the test timeout, mark the AC as FAILED and capture a screenshot of the current desktop state.
Fallback: If a specific AC cannot be translated into an automated test (e.g., it requires human aesthetic judgment), mark it as MANUAL CHECK REQUIRED and continue.
Phase 5: Map Results to Acceptance Criteria
Goal: Compare every test execution result against the [IMP] acceptance criteria from the functional specification and produce a deterministic verdict for each.
5.1 Initialize the Results Table
Create an associative results structure (e.g., shell associative array, JSON object, or temporary file) keyed by AC ID. For every [IMP] acceptance criterion extracted in Phase 1, pre-populate a row with:
Field
Initial Value
ac_id
The AC identifier (e.g., AC-012)
ac_text
The full criterion text, truncated to 120 characters for display
status
PENDING (updated in 5.2)
evidence
Empty string (updated in 5.2)
actual
Empty string (populated only on mismatch)
expected
Empty string (populated only on mismatch)
Iterate over the specification’s acceptance criteria table and include only rows whose taxonomy tag is [IMP]. [SEF] and [EXT] criteria are excluded from runtime verification; they may be listed in an appendix but do not require a status verdict.
5.2 Apply Verdict Rules
For each AC that was targeted by a test in Phase 4, apply the following rules in order:
VERIFIED (REQ-016):
IF the test completed without assertion failures, timeouts, or tool errors
THEN set status = "VERIFIED"
Set evidence to a concise description:
REST API: curl returned HTTP <code> in <N>ms
Web SPA: Playwright assertion passed: <selector> <condition>
IF any assertion failed, the test timed out, the tool crashed, or the dev server was unreachable
THEN set status = "FAILED"
Set actual to the observed value (e.g., HTTP 400, element not found, timeout after 30s)
Set expected to the value demanded by the AC (e.g., HTTP 201, element visible)
Set evidence to a human-readable sentence combining actual vs expected, plus the path to any captured artifact (screenshot, response dump) if available.
MANUAL CHECK REQUIRED (REQ-017):
IF no test was generated for the AC because:
The AC text lacked parseable endpoint / UI / desktop behavior hints
A required tool was missing (e.g., Playwright not installed)
The AC describes human judgment (e.g., "UI looks correct")
THEN set status = "MANUAL CHECK REQUIRED"
Set evidence to the reason: No automated test could be derived: <reason>
IMPORTANT: Every [IMP] AC MUST have a final status of VERIFIED, FAILED, or MANUAL CHECK REQUIRED. No AC may remain in PENDING when Phase 5 ends.
## Environment
| Property | Value |
|----------|-------|
| Application Type | ${APP_TYPE} |
| Startup Command | \`${STARTUP_COMMAND}\` |
| Target Port | ${TARGET_PORT} |
| Health Endpoint | ${HEALTH_CHECK_METHOD} |
| Runtime Version | ${RUNTIME_VERSION} |
| Verification Started | ${ISO8601_START_TIME} |
RUNTIME_VERSION: capture the runtime version detected during startup:
Docker: docker --version
Node.js: node --version
Java: java -version 2>&1 | head -1
Python: python --version or python3 --version
ISO8601_START_TIME: the timestamp when Phase 3 began, in ISO-8601 format.
Section 3 — Test Results
## Test Results
| AC ID | Criterion (truncated) | Status | Evidence |
|-------|----------------------|--------|----------|
| AC-012 | WHEN the app is classified... | VERIFIED | \`curl\` returned 200 |
| AC-013 | WHEN a curl test executes... | FAILED | Expected 201, got 400 |
| AC-014 | WHEN authentication is required... | MANUAL CHECK REQUIRED | No credentials found |
The "Criterion (truncated)" column MUST be truncated to 120 characters, appended with ... if longer.
The "Status" column MUST be one of VERIFIED, FAILED, or MANUAL CHECK REQUIRED.
The "Evidence" column MUST reference artifacts using relative paths when applicable (e.g., ./e2e-artifacts/ac-019-screenshot.png).
Section 4 — Raw Output
## Raw Output### Startup Logs
\`\`\`
[contents of $STARTUP_LOGS_FILE.redacted_final]
\`\`\`
### Test Commands
\`\`\`
[For each test: the exact command executed (auth redacted) and its first 2000 chars of stdout/stderr]
\`\`\`
If a test produced no output, write (no output).
Limit each code block to 2000 lines to avoid overwhelming the report; if logs are larger, append a note: (truncated; full logs available in <path>).
Section 5 — Artifacts
## Artifacts
| AC ID | Type | Path |
|-------|------|------|
| AC-019 | Screenshot | ./e2e-artifacts/screenshot-1717189200-ac-019.png |
| AC-014 | Response Dump | ./e2e-artifacts/ac-014-response.json |
List every artifact file present in $ARTIFACT_DIR.
If no artifacts were generated, write No artifacts captured for this run.
Section 6 — Teardown Status
## Teardown Status
| Property | Value |
|----------|-------|
| Teardown Executed | ${TEARDOWN_EXECUTED} |
| Port Released | ${PORT_RELEASED} |
| Remaining Processes | ${REMAINING_PROCESSES} |
The report MUST be valid Markdown (passes basic markdownlint rules): proper heading levels, no trailing spaces, consistent pipe table delimiters.
The report MUST NOT be modified after it is written (REQ-NR005).
Set a variable REPORT_PATH="$REPORT_FILE" for use in Phase 8.
Phase 7: Teardown Environment
Goal: Gracefully stop all runtime resources started in Phase 3, verify that ports and processes are fully released, and record the outcome. If --keep-alive is passed, skip teardown entirely and warn the user.
7.1 --keep-alive Guard
if [ "$KEEP_ALIVE" = "true" ]; then
TEARDOWN_EXECUTED="skipped (--keep-alive)"
PORT_RELEASED="skipped"
REMAINING_PROCESSES="skipped"echo"WARNING: --keep-alive was passed. The runtime environment is still running."echo"You are responsible for cleaning up:"if [ "$APP_TYPE" = "Docker-managed" ]; thenecho" docker compose -f '${PROJECT_ROOT}/docker-compose.yml' down"elseecho" kill $(lsof -ti:${TARGET_PORT})"fiecho"Report was saved before teardown; artifacts are in ${ARTIFACT_DIR}."# Skip remaining teardown steps and proceed to Phase 8# (Phase 7 cleanup is bypassed when --keep-alive is active)fi
7.2 Graceful Teardown by Application Type
Docker-managed (AC-029):
TEARDOWN_EXECUTED="true"
DOCKER_TEARDOWN_FAILED=falseif [ "$APP_TYPE" = "Docker-managed" ]; thencd"$PROJECT_ROOT"if docker compose down >> "$STARTUP_LOGS_FILE" 2>&1; thenecho"Docker Compose stack stopped successfully."elseecho"WARNING: docker compose down exited with a non-zero code."
DOCKER_TEARDOWN_FAILED=truefifi
Local processes (JVM, Node.js/NestJS, Python, Desktop) (AC-029):
if [ -n "$STARTUP_PID" ] && kill -0 "$STARTUP_PID" 2>/dev/null; thenecho"Sending SIGTERM to process $STARTUP_PID ..."kill -TERM "$STARTUP_PID" 2>/dev/null || true# Wait up to 10 seconds for graceful exit
GRACEFUL_WAIT=0
while [ "$GRACEFUL_WAIT" -lt 10 ] && kill -0 "$STARTUP_PID" 2>/dev/null; dosleep 1
GRACEFUL_WAIT=$((GRACEFUL_WAIT + 1))
done# If still alive, send SIGKILLifkill -0 "$STARTUP_PID" 2>/dev/null; thenecho"Process $STARTUP_PID did not exit after 10s; sending SIGKILL."kill -KILL "$STARTUP_PID" 2>/dev/null || truesleep 2
fifi# Also terminate any child processes that may still hold the port
PORT_PIDS=$(lsof -ti:"$TARGET_PORT" 2>/dev/null || true)
if [ -n "$PORT_PIDS" ]; thenecho"Additional PIDs holding port $TARGET_PORT: $PORT_PIDS"for pid in$PORT_PIDS; dokill -TERM "$pid" 2>/dev/null || truedonesleep 3
for pid in$PORT_PIDS; doifkill -0 "$pid" 2>/dev/null; thenkill -KILL "$pid" 2>/dev/null || truefidonefi
Emergency cleanup — orphan processes from a crashed test:
# If the original STARTUP_PID is empty or already dead, but the port is still occupied,# this indicates an orphan process from a crash or a detached child.if [ -z "$STARTUP_PID" ] || ! kill -0 "$STARTUP_PID" 2>/dev/null; then
ORPHAN_PIDS=$(lsof -ti:"$TARGET_PORT" 2>/dev/null || true)
if [ -n "$ORPHAN_PIDS" ]; thenecho"WARNING: Orphan process(es) detected on port $TARGET_PORT: $ORPHAN_PIDS"for pid in$ORPHAN_PIDS; dokill -TERM "$pid" 2>/dev/null || truedonesleep 3
for pid in$ORPHAN_PIDS; doifkill -0 "$pid" 2>/dev/null; thenkill -KILL "$pid" 2>/dev/null || truefidonefifi
7.3 Verify Port Release (AC-030)
PORT_RELEASED="false"
PORT_CHECK_WAIT=0
MAX_PORT_CHECK_WAIT=15
while [ "$PORT_CHECK_WAIT" -lt "$MAX_PORT_CHECK_WAIT" ]; doif ! lsof -i :"$TARGET_PORT" >/dev/null 2>&1 && ! nc -z localhost "$TARGET_PORT" 2>/dev/null; then
PORT_RELEASED="true"echo"Port $TARGET_PORT is free."breakfisleep 1
PORT_CHECK_WAIT=$((PORT_CHECK_WAIT + 1))
doneif [ "$PORT_RELEASED" != "true" ]; thenecho"WARNING: Port $TARGET_PORT is still occupied after teardown."fi
7.4 Identify Remaining Resources (REQ-023)
REMAINING_PROCESSES="none"if [ "$APP_TYPE" = "Docker-managed" ]; then
REMAINING_CONTAINERS=$(docker compose ps --format '{{.Name}}' 2>/dev/null | tr'\n'',' | sed 's/,$//')
if [ -n "$REMAINING_CONTAINERS" ]; then
REMAINING_PROCESSES="containers: $REMAINING_CONTAINERS"fielse
REMAINING_PIDS=$(lsof -ti:"$TARGET_PORT" 2>/dev/null | tr'\n'' ' | sed 's/ $//')
if [ -n "$REMAINING_PIDS" ]; then
REMAINING_PROCESSES="PIDs: $REMAINING_PIDS"fifi
7.5 Warn User on Incomplete Teardown
if [ "$PORT_RELEASED" != "true" ] || [ "$DOCKER_TEARDOWN_FAILED" = "true" ]; thenecho"WARNING: Teardown did not complete cleanly. Manual cleanup may be required."if [ "$APP_TYPE" = "Docker-managed" ]; thenecho" Remaining containers: ${REMAINING_PROCESSES}"echo" Manual command: docker compose -f '${PROJECT_ROOT}/docker-compose.yml' down"elseecho" Remaining PIDs on port ${TARGET_PORT}: ${REMAINING_PIDS}"echo" Manual command: kill ${REMAINING_PIDS}"fifi
7.6 Record Teardown Status
Update the report file in-place (append the Teardown Status section if it was not written in Phase 6, or ensure the variables are correctly set before Phase 6 finalizes). The preferred order is:
Phase 6 writes the report with placeholder teardown variables.
Phase 7 executes teardown.
Phase 7 updates the placeholders with actual values using an in-place edit (e.g., sed):
Non-whitelisted command policy: IF a derived command is NOT in the whitelist → use AskUserQuestion to request explicit user confirmation before execution. The user MUST approve the command; otherwise, abort.
Security Guards and Negative Requirements
The following negative requirements (REQ-NR001 through REQ-NR008) are enforced at multiple points in the skill workflow:
REQ-NR001: No Destructive Commands
The system SHALL NOT run sudo, rm -rf, docker system prune, mkfs, dd, or any other destructive system command during startup, test, or teardown.
Enforcement:
Phase 1.5 scans all commands for forbidden patterns before execution.
Phase 3 data integrity pre-flight rejects commands that would destroy data.
Teardown commands are restricted to docker compose down and graceful process termination (kill -TERM / kill -KILL for stuck orphans only).
REQ-NR002: No Secrets in Report
The system SHALL NOT expose secrets, API keys, or passwords in the E2E report.
IF --insecure is passed → a warning is logged: "TLS certificate verification is disabled. Use only for local development."
REQ-NR004: No Data Overwrite on Startup
The system SHALL NOT overwrite or delete existing databases, volumes, or local data unless the startup command itself does so.
Enforcement:
Phase 1.5 data integrity pre-check rejects commands containing rm, --volumes, prune, drop, or similar destructive data patterns.
Phase 3 data integrity pre-flight aborts startup if the command would destroy existing data.
REQ-NR005: No Spec or Task File Modification
The system SHALL NOT modify the functional specification or task files during report generation or teardown.
Enforcement:
Phase 6 writes the report as a new file (e2e-report-YYYY-MM-DD-HHMMSS.md) only.
Phase 7 teardown appends only a Post-Teardown Update section to the report; it never modifies spec or task files.
The skill is read-only regarding source code, specs, and tasks.
REQ-NR006: No Orphan Processes on Failure
The system SHALL NOT leave orphan processes or containers running when startup fails.
Enforcement:
Phase 3 Docker startup: on timeout, docker compose down is executed before exiting.
Phase 3 Spring Boot / Node.js startup: on timeout, kill -TERM $STARTUP_PID is executed before exiting.
Phase 7 teardown includes emergency orphan cleanup for processes still holding the target port.
REQ-NR007: Per-Test Timeout Enforcement
The system SHALL NOT block indefinitely on a hanging test.
Enforcement:
Phase 4 Playwright tests: wrapped with timeout --signal=TERM <N>; if timeout occurs, the AC is marked FAILED.
Phase 4 Desktop tests: a manual polling loop checks elapsed time against TEST_TIMEOUT_SEC; if exceeded, the AC is marked FAILED.
If a test hangs, it is terminated and marked FAILED, never VERIFIED or PENDING.
REQ-NR008: No Retry on Failed curl Tests
The system SHALL NOT retry failed curl requests.
Enforcement:
Phase 4 curl execution: each command is executed exactly once.
NO --retry flag, NO retry loops, NO fallback re-execution.
The first result is recorded immediately; if it fails, the AC is marked FAILED.
Constraints and Warnings
Read-only on source: This skill never edits application source code, test files, configuration, functional specifications, or task files.
No auto-install: If Playwright or Docker is missing, the skill reports the gap and suggests install commands. It does NOT run npm install -g or brew install automatically.
Local only: Tests run against localhost. Remote URLs, staging, or production endpoints are out of scope.
Destructive guard: Startup, test, and teardown commands are whitelisted per references/test-execution-patterns.md. Any command outside the whitelist requires user confirmation via AskUserQuestion.
Secret hygiene: Authorization headers, bearer tokens, E2E_* environment variables, and passwords are redacted from the report. Only header names and redacted values appear.
TLS default: curl commands use strict TLS verification by default. Pass --insecure only for local development with self-signed certificates.
Cleanup responsibility: If --keep-alive is passed, the user is responsible for manual teardown.
No persistent test suite: The skill generates ad-hoc tests for verification. It does not create permanent test files in the project.
head
Fallback defaults (if no port is found in any config file):
App Type
Default Port
Node.js / NestJS
3000
Spring Boot (JVM)
8080
Angular
4200
Vite (React/Vue)
5173
Python
8000
done
if
"$READY"
"true"
then
echo
"Startup timeout (${STARTUP_TIMEOUT}s) exceeded for Docker Compose."
tail
"$STARTUP_LOGS_FILE"
# Attempt cleanup to avoid orphan containers (REQ-NR006)
true
exit
fi
if
"$TARGET_PORT"
then
true
"port polling via nc on localhost:${TARGET_PORT} (every 3s)"
break
fi
# Fail fast if the background process exited early
if
"$STARTUP_PID"
kill
"$STARTUP_PID"
then
echo
"Spring Boot process exited before reaching healthy state."
break
fi
sleep
done
if
"$READY"
"true"
then
echo
"Startup timeout (${STARTUP_TIMEOUT}s) exceeded for Spring Boot."
# Capture last lines of startup logs for the report
tail
"$STARTUP_LOGS_FILE"
"$STARTUP_LOGS_FILE"
true
# Attempt cleanup (REQ-NR006)
"$STARTUP_PID"
kill
"$STARTUP_PID"
true
exit
fi
"port polling via nc/curl on localhost:${TARGET_PORT} (every 3s)"
false
while
"$STARTUP_TIMEOUT"
do
# Option A: HTTP readiness via curl
if
"http://localhost:${TARGET_PORT}"
then
true
"curl on localhost:${TARGET_PORT}"
break
fi
# Option B: Raw port readiness via nc
if
"$TARGET_PORT"
then
true
"nc -z localhost:${TARGET_PORT}"
break
fi
# Fail fast if the background process exited early
if
"$STARTUP_PID"
kill
"$STARTUP_PID"
then
echo
"Node.js/NestJS process exited before reaching healthy state."
break
fi
sleep
done
if
"$READY"
"true"
then
echo
"Startup timeout (${STARTUP_TIMEOUT}s) exceeded for Node.js/NestJS."
tail
"$STARTUP_LOGS_FILE"
"$STARTUP_LOGS_FILE"
true
# Attempt cleanup (REQ-NR006)
"$STARTUP_PID"
kill
"$STARTUP_PID"
true
exit
fi
Console ready detection: After readiness is confirmed, scan $STARTUP_LOGS_FILE for common server-ready messages (e.g., Nest application successfully started, Local:, ready in, Server running) and record the first matching line in the report as evidence of successful startup.
if
"$BUILD_EXIT_CODE"
then
echo
"Tauri build failed (exit code $BUILD_EXIT_CODE). See startup logs for details."
exit
fi
"$BUILD_COMMAND (build succeeded)"
fi
Electron:
if [ "$DESKTOP_FRAMEWORK" = "electron" ]; thenif [ ! -d "$PROJECT_ROOT/node_modules" ]; thenecho"ERROR: node_modules not found. Run 'npm install' before verification."exit 1
fi
BUILD_COMMAND="(no separate build required for dev mode)"
STARTUP_COMMAND="npm run electron:dev"if grep -q '"electron:build"'"$PROJECT_ROOT/package.json" 2>/dev/null; then
BUILD_COMMAND="npm run electron:build"cd"$PROJECT_ROOT" && $BUILD_COMMAND >> "$STARTUP_LOGS_FILE" 2>&1
BUILD_EXIT_CODE=$?
if [ "$BUILD_EXIT_CODE" -ne 0 ]; thenecho"Electron build failed (exit code $BUILD_EXIT_CODE). See startup logs for details."exit 1
fififi
