| name | journey-test-harness |
| description | Run multiple journeys as a cross-platform test suite. Discover journeys, invoke journey-runner in isolated workspaces, deploy, verify, capture screenshots, clean up only owned Azure resources, and produce a consolidated report.
USE FOR: test all journeys, regression test, CI journey validation, nightly journey test, end-to-end journey suite, validate journeys deploy correctly.
DO NOT USE FOR: running one journey interactively (use journey-runner), creating journeys (use journey-template), or reviewing content (use content-reviewer).
|
Journey Test Harness
Orchestrate journey-runner across the selected journey directories. The harness must work on Windows, macOS, and Linux and must never delete an Azure environment merely because it appears in azd env list.
Inputs
- Journeys, default
all
- Stack, default Node.js/TypeScript for multi-stack journeys
- Deploy, default
true
- Location, default
westus
- Concurrency, default
1; increase only when quota, local ports, and API limits allow it
Each deployed journey uses cleanup: after-verification unless the user explicitly requests otherwise.
Step 1: Discover Journeys
Use repository file-search APIs or Node.js fs.readdir() to find journeys/*/README.md. Do not use ls | sed, shell globs, or platform-specific path parsing.
For each journey, record:
- Folder name and absolute source path
- Journey type
PLAN.md presence- Required and optional tools from the journey prerequisite section
- Estimated cost and time
- Screenshot requirement
Filter only against the discovered folder names. Fail early for an unknown requested journey.
Step 2: Suite Preflight
Load journey-runner and run its cross-platform preflight before creating any workspace or Azure resource.
The union of selected-journey requirements may include:
- Node.js 24 LTS or later
- Azure CLI and valid authentication
azd 1.28.0 or later with auth.useAzCliAuth=true- GitHub Copilot CLI
- Docker daemon and Buildx for AIMarket
kubectl and Helm 3 for Superset- Azure Functions Core Tools v4, Azurite, and Go-based
sqlcmd for SmartTodo
- The pinned Playwright package and bundled Chromium for web screenshots
- Xcode 16+ only when SmartTodo iOS execution is requested on macOS
Use .github/skills/journey-runner/scripts/check-prerequisites.mjs with the union of required tools. Missing required tools stop the whole suite before provider registration. Do not install system tools during the suite.
Step 3: Azure Preparation
Check Azure CLI and azd authentication separately. Confirm the intended subscription and location.
Register only providers required by selected journeys. Run provider commands as individual processes with argument arrays, not shell loops. Wait for required registrations before starting the first deployment.
Record the subscription ID and each provider's state in the suite report.
Step 4: Create Suite and Journey Workspaces
Create directories through Node.js filesystem APIs or the active agent's file tools:
<journey-runs-root>/test-suite-<UTC timestamp>/
├── test-report.md
├── screenshots/
└── runs/
└── <journey>-<UTC timestamp>/
Use path.join() and absolute paths. Don't embed ~/, date, mkdir, or shell-specific separators in executable instructions.
Copy PLAN.md with fs.copyFile() when needed. Generated application code, infrastructure, logs, and secrets stay inside the isolated run directory, never the source repository.
Step 5: Run Each Journey
Invoke journey-runner with:
Journey: <journey-source-path>
Stack: <stack>
Location: <location>
Working directory: <absolute-run-directory>
Cleanup: after-verification
The runner owns per-journey preflight, prompts, local ports, deployment, verification, screenshots, and scoped cleanup.
Run serially by default. Parallel runs require:
- Distinct local ports
- Distinct
azd environment names
- Distinct resource groups
- Sufficient regional quota
- Independent log and screenshot paths
Never share an azd environment across journeys.
Step 6: Preserve Evidence
Before removing a run directory, copy these artifacts with Node.js filesystem APIs:
run-report.mdissues.mdscreenshot-*.png- Sanitized deployment and verification logs
Use the mapping below:
| Journey | Web output | Screenshot |
|---|
| Grafana | GRAFANA_URL | Login page |
| n8n | N8N_URL | Owner-setup or login page |
| Superset | SUPERSET_URL | Authenticated welcome page when credentials are available |
| AIMarket | WEB_URL | Storefront with all images loaded |
| SmartTodo | None | No screenshot on Windows/Linux; simulator screenshot only on a suitable macOS/Xcode host |
Use journey-runner's pinned Playwright Chromium helper. Never use a branded Chrome channel.
Step 7: Guaranteed Scoped Cleanup
Cleanup runs in a finally path after each deployment attempt, including build, deployment, verification, or screenshot failures.
For each journey, delete only:
- Its explicitly recorded
azd environment
- Its explicitly recorded resource groups and managed resource groups
- Soft-deleted resources whose names were recorded as owned by that run
Never run a pipeline over every row from azd env list. Other environments may belong to unrelated work.
After azd down --force --purge --no-prompt, verify:
- Recorded resource groups return not found
- No active resource remains with that run's environment tag
- Any owned soft-deleted resource selected for purge is absent
- Unrelated environments and resource groups remain unchanged
If cleanup is incomplete, preserve the workspace and list exact remaining resource IDs in the consolidated report.
Step 8: Workspace Retention
Delete a per-journey working directory only when all of these are true:
- Evidence was copied successfully.
- Azure cleanup was live-verified.
- The user didn't request generated code retention.
- No unresolved cleanup or debugging artifact depends on the directory.
Use Node.js fs.rm(path, { recursive: true, force: true }) on the exact recorded run directory. Never construct a deletion path from untrusted output.
Step 9: Consolidated Report
Write test-report.md directly with file tools or Node.js. Don't build Markdown through a large shell one-liner.
Include:
- UTC start/end and total duration
- Host OS and architecture
- Tool versions
- Subscription and location
- Selected journeys and stack
- Build, deployment, verification, screenshot, and cleanup status per journey
- Preserved artifact paths
- Exact blockers and remaining resource IDs
Example result table:
| Journey | Build | Deploy | Verify | Screenshot | Cleanup | Result |
|---|
| Grafana | PASS | PASS | PASS | PASS | PASS | PASS |
| n8n | PASS | PASS | PASS | PASS | PASS | PASS |
| Superset | PASS | PASS | PASS | PASS | PASS | PASS |
| AIMarket | PASS | PASS | PASS | PASS | PASS | PASS |
| SmartTodo | PASS | PASS | PASS | N/A on Linux/Windows | PASS | PASS |
A journey can't receive PASS when cleanup was requested but not verified.
Failure Policy
- Missing prerequisite: stop before Azure work.
- Build failure: one evidence-based repair, then mark FAIL.
- Deployment failure: collect diagnostics, then run scoped cleanup.
- Verification failure: retry only documented transient cases, then mark FAIL.
- Screenshot failure: preserve logs and continue to cleanup; mark PARTIAL only if all functional checks passed.
- Cleanup failure: mark FAIL and preserve the run directory.
Safety Rules
- Never install system packages or privileged emulation automatically.
- Never delete all
azd environments.
- Never broaden cleanup scope after a failed deletion.
- Never expose credentials, cookies, authorization headers, connection strings, or generated secrets.
- Never claim macOS, Windows, iOS, or browser verification that didn't actually run.
