| name | sat-run |
| description | Execute SAT test scenarios against the real Branchdeck app via WebDriver. Reads scenarios from sat/scenarios/, runs each against a debug binary using tauri-driver + WebdriverIO, captures screenshots and trajectory data. Use when user says "run sat scenarios", "sat run", "execute scenarios", or "test the app with sat".
|
SAT Scenario Runner
You execute SAT test scenarios against the real Branchdeck application using WebDriver automation. Each scenario's natural-language steps are translated into browser actions, with before/after screenshots and performance metrics captured per step.
Prerequisites
Before running, verify these are in place:
-
Debug binary exists — Check if src-tauri/target/debug/branchdeck exists. If not, tell the user:
"Debug binary not found. Building now..." and run:
bunx tauri build --debug --no-bundle
-
tauri-driver installed — Check if ~/.cargo/bin/tauri-driver exists. If not:
"tauri-driver not found. Install with: cargo install tauri-driver --locked"
-
WebKitWebDriver available — Check if WebKitWebDriver is in PATH. If not:
"WebKitWebDriver not found. Install with: sudo apt install webkit2gtk-driver"
-
Scenarios exist — Check if sat/scenarios/*.md files exist (exclude .gitkeep and .sat-state.yaml). If none:
"No scenarios found. Run /sat-generate first."
Arguments
The user may provide:
- scenario — path to a single scenario file to run (default: run all scenarios)
- limit — max number of scenarios to run (default: all)
Step 1: Setup Run Directory
Create the output directory:
RUN_DIR="sat/runs/run-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$RUN_DIR/screenshots"
Step 2: Execute Scenarios
For each scenario file (or the single specified one):
Run the scenario via the SAT WebdriverIO bridge:
SAT_SCENARIO_FILE="<scenario-path>" SAT_RUN_DIR="<run-dir>" \
xvfb-run bunx wdio run sat/scripts/wdio.sat.conf.ts 2>&1
This will:
- Start
tauri-driver
- Launch the debug binary in the WebKitGTK webview
- Wait for the app to load ("Branchdeck" title)
- Execute each scenario step:
- Take before screenshot
- Interpret the natural-language step and execute via WebDriver
- Take after screenshot
- Record result in trajectory
- Write
trajectory-{scenario-id}.json to the run directory
- Kill the session
Error handling per scenario:
- If the WebdriverIO run exits non-zero, check the trajectory file — partial results are still written
- If no trajectory file was created, log the error and continue to the next scenario
- Each scenario gets its own WebDriver session (app restarts fresh)
Step 3: Review Results
After all scenarios complete, read each trajectory-*.json file from the run directory and compile a summary.
For each trajectory, report:
- Scenario ID and title
- Status (completed / execution_failed)
- Steps passed vs failed
- Total duration
- Any failure reasons
Aggregate metrics:
- Total scenarios run
- Pass rate (completed scenarios / total)
- Average step success rate
- Total screenshots captured
- Total run duration
- Memory usage
Present as a table:
| Scenario | Status | Steps | Passed | Failed | Duration |
|----------|--------|-------|--------|--------|----------|
| ... | ... | ... | ... | ... | ... |
Step 4: Performance Report
Read performance data from each trajectory and report:
- Per-step timing — average, min, max across all steps
- Per-scenario timing — total duration per scenario
- Memory usage — heap MB at end of each scenario
- Screenshot count — total before + after screenshots
Common Issues
"Cannot find module 'run-scenario.ts'"
The SAT scripts need the project's WebdriverIO types. Run bun install first.
"Debug binary not found"
The onPrepare hook checks for the binary. Build with:
bunx tauri build --debug --no-bundle
"Connection refused on port 4444"
tauri-driver failed to start. Check if it's installed:
which tauri-driver
Session dies mid-scenario
The app may have crashed. Check src-tauri/target/debug/branchdeck is a recent build. The trajectory file will still be written with partial results — remaining steps are marked as failed.
