woo-ai-smoke

# woo-ai-smoke — evaluation methodology This skill evaluates the `WooAIAssistant` feature beyond surface smoke. It combines **hard invariants** (deterministic, must-hold) with a **rubric scored by Claude** across four dimensions (correctness, groundedness, tool appropriateness, recovery). Runs are stored append-only under `runs/` so regressions over commits are detectable. ## Delegation model (MANDATORY) **Main Claude never runs the pipeline itself.** A single smoke run ingests ~70+ [smoke|...] lines plus thousands of xcodebuild log lines — that's a context firehose. Instead: 1. Main Claude parses `$ARGUMENTS` (suite/scenario/samples/mode) and picks the baseline to compare against. 2. Main Claude dispatches a **single subagent via the Task tool** with the self-contained prompt below. Use `subagent_type: "general-purpose"` so the subagent has full tool access (Bash, Read, Write, Edit, Grep, Glob). 3. The subagent does everything in the "Full execution checklist" — credential refresh, writing the Swift template, xcodebuild run, parsing, judging, JSONL write, cleanup. 4. The subagent returns **only** the final markdown report: the per-scenario table + the PASS/REGRESSION/FAIL/NEW/FLAKY summary + the JSONL run path + 1-2 lines per regression. 5. Main Claude relays that markdown verbatim to the user. Do not re-judge, re-parse, or re-print raw [smoke|...] lines in the main context. ### Subagent prompt template Fill in the placeholders (in ALL CAPS) before dispatching: ``` You are running the /woo-ai-smoke pipeline end-to-end. Follow the SKILL.md at .claude/skills/woo-ai-smoke/SKILL.md as your reference for the Swift template, parse protocol, hard invariants, rubric, JSONL format, and reporting format. Everything below is your SCOPED task. Inputs: - mode: rest # only "rest" is wired up; MCP support is deferred - suite: SUITE # "default" (24 scenarios × N samples) or ad-hoc "t1; t2" - samples: N # 1 for ad-hoc, 3 for default - baseline: BASELINE # path to baseline JSONL to compare against - run_label: LABEL # short tag for the stored run file, e.g. "post_prompt_revision" - head_sha: SHA # from `git rev-parse --short HEAD` - branch: BRANCH # from `git branch --show-current` Pipeline (execute in this order, no skipping). Arm a `trap` cleanup at the start so a build crash never leaves the temp Swift file or log behind: ```bash trap 'rm -f Modules/Tests/WooAIAssistantTests/SmokeRespondContractTests.swift /tmp/woo-ai-smoke.log /tmp/woo-ai-smoke-store.env' EXIT ``` 1. Verify ~/.woo-ai-smoke/store.env exists with all five required keys (WOO_SITE_URL, WOO_SITE_ID, WOO_USERNAME, WOO_APP_PASSWORD, WOO_DOTCOM_ACCESS_TOKEN). On first run the file doesn't exist: scaffold it with placeholders per the Credentials section of SKILL.md, `open` it for editing, and stop with a message instructing the engineer to fill it in and re-run. 2. Load the scenario set from .claude/skills/woo-ai-smoke/baseline.json (or build ad-hoc from the SUITE arg). 3. Run the Scenario fixture preflight from SKILL.md for exactly the scenarios being executed. Inspect each scenario's `fixtures` block first, then infer obvious missing fixtures from the prompts/rubric. Use the WooCommerce REST API with the smoke credentials to verify fixtures exist and create/update only smoke-owned records when needed. If a required fixture cannot be created, stop before xcodebuild with a short fixture error report. 4. Write Modules/Tests/WooAIAssistantTests/SmokeRespondContractTests.swift from the template in SKILL.md, replacing SAMPLES_PLACEHOLDER with N and wiring each scenario's turns and derived autoDeclineWrites (default true when scenario.category == "write" or the scenario has any write tools in its hard invariants, false otherwise — unless the baseline scenario explicitly sets autoDeclineWrites on the turn). Mode "rest" uses the default WooAssistantHeadless tool source. The Swift template's specific API references (resolver typealias, Card.kind shape) may drift between trunk states — if the build fails on them, fix inline in the generated test file rather than the template. 5. Run xcodebuild with the command in SKILL.md's Running section. Tee full output to /tmp/woo-ai-smoke.log. You may run it in the background and poll the log, but you must wait for completion before parsing. 6. Parse every [smoke|...] line per SKILL.md Parse protocol. 7. Apply hard invariants deterministically. A hard-invariant failure is an automatic FAIL; do not rubric-score further. 8. For every remaining turn, judge yourself against the rubric in SKILL.md plus the scenario's rubric_notes from baseline.json. Score 0/1/2 per dim, write a one-sentence rationale. 9. Compute per-scenario means (over samples × turns) per dim. 10. Write the run to .claude/skills/woo-ai-smoke/runs/<ISO-timestamp>_SHA_LABEL.jsonl (one JSON record per turn per sample per mode, exactly as defined in SKILL.md Storage format). 11. Compare against BASELINE: classify each scenario PASS / REGRESSION / FAIL / NEW / FLAKY per the Outcome classification table. 12. Cleanup is automatic via the `trap` armed at step 0; verify the three artifacts are gone before returning. Return ONLY this markdown (no tool logs, no chain-of-thought, no raw [smoke|...] lines). Main Claude will relay this verbatim: ## Smoke result — MODE vs BASELINE <the markdown table from SKILL.md Reporting section, one row per scenario> PASS: X | REGRESSION: Y | FAIL: Z | NEW: W | FLAKY: V Run stored: .claude/skills/woo-ai-smoke/runs/<filename>.jsonl <one or two lines per REGRESSION / FAIL with likely cause> If the build fails or a hard harness error halts the run, return the short error + what you cleaned up, not a full log dump. ``` Keep the subagent dispatch in a single Task tool call. Never split the pipeline into multiple subagent turns — the parse state has to stay inside the subagent's context. ## How it works 1. **Load scenarios** — default suite (24 scenarios) from `baseline.json`, or ad-hoc via `scenario "turn1; turn2"`. 2. **Verify credentials** in `~/.woo-ai-smoke/store.env` (see "Credentials" below). Swift reads the dotenv directly each run. 3. **Preflight fixtures** for the selected scenarios. Verify/create smoke-owned products, orders, and customers through the WooCommerce REST API before running the model. 4. **Write `Modules/Tests/WooAIAssistantTests/SmokeRespondContractTests.swift`** using the template below. Scenarios get expanded into the `@Test(arguments:)` parametrised suite. 5. **Run the smoke via `xcodebuild`**, capture stdout. 6. **Parse each `[smoke|...]` line** into a turn record — prompt, tool names, tool arg snippets, tool results, assistant text, card kinds. 7. **Claude judges each turn** against the scenario's `rubric_notes` and the global rubric (details below). Fill in scores per dim. 8. **Apply hard invariants** (deterministic pass/fail). 9. **Write run** to `.claude/skills/woo-ai-smoke/runs/<ISO-timestamp>_<sha>.jsonl`. 10. **Compare to baseline** — flag REGRESSION when hard invariants fail or rubric mean drops below `rubric_pass_threshold`. 11. **Report** a markdown table + summary counts. 12. **Delete** the temp Swift file, `/tmp/woo-ai-smoke.log`, and the `/tmp/woo-ai-smoke-store.env` mirror (via the `trap` armed at the start of the run). ## Prerequisites - Xcode + iOS simulator (the project's `bootstrap` skill covers this). - A WooCommerce demo store with an admin **application password** (for the REST tool calls) and an authenticated iOS app session whose WPCOM OAuth bearer can be captured (for the woo-mobile-ai LLM calls). - Required CLI tools (all macOS-default): `xcodebuild`, `xcrun simctl`, `open`. - Store credentials in **`~/.woo-ai-smoke/store.env`** with `WOO_SITE_URL`, `WOO_SITE_ID`, `WOO_USERNAME`, `WOO_APP_PASSWORD`, and `WOO_DOTCOM_ACCESS_TOKEN`. On first run the skill scaffolds the file with placeholders and opens it for editing — see Credentials below. The skill never commits credentials. Swift reads `~/.woo-ai-smoke/store.env` directly so nothing leaks to `/tmp`. ## Credentials The engineer maintains `~/.woo-ai-smoke/store.env` (the source of truth, dotenv format). The skill stages a `/tmp/woo-ai-smoke-store.env` mirror at run-start because the iOS simulator process sandboxes `~` to its own container and can't read the host's home directly; the `trap` cleanup deletes the `/tmp` mirror at run-end. Swift reads from `/tmp/woo-ai-smoke-store.env`. The harness sends LLM traffic through the wpcom `woo-mobile-ai` backend wrapper using a captured iOS-app WPCOM OAuth bearer (`WOO_DOTCOM_ACCESS_TOKEN`). For pre-merge testing the engineer can route locally via mitmproxy, `/etc/hosts`, or a temporary hardcoded URLSession in the harness (not committed); the committed code only ships production-URL routing because nginx on the wpcom sandbox vhost rejects requests whose `Host` header isn't `public-api.wordpress.com`. REST tool calls still hit the merchant store directly with the application password. **First-run flow**: if `~/.woo-ai-smoke/store.env` doesn't exist, scaffold it with placeholders, open it for the engineer to fill in, then stop. The engineer saves the file and re-runs the skill. ```bash ENV_FILE="$HOME/.woo-ai-smoke/store.env" STAGED_ENV="/tmp/woo-ai-smoke-store.env" # First run: scaffold the file with placeholders, open it for editing, stop. if [ ! -f "$ENV_FILE" ]; then mkdir -p "$(dirname "$ENV_FILE")" cat > "$ENV_FILE" <<'TEMPLATE' # Woo AI smoke credentials - fill these in, save, then re-run the smoke skill. # WOO_SITE_ID is the WordPress.com blog id of the demo store. Find it in # wp-admin/options-general.php?page=jetpack or via the Jetpack AI JWT mint. WOO_SITE_URL=https://your-demo-store.example.com WOO_SITE_ID=123456 WOO_USERNAME=your-admin-username WOO_APP_PASSWORD=xxxx xxxx xxxx xxxx xxxx xxxx # WPCOM OAuth bearer captured from an authenticated iOS app session. Required # for the woo-mobile-ai LLM path. Grab it by inspecting any /me request the # app issues. WOO_DOTCOM_ACCESS_TOKEN= TEMPLATE chmod 600 "$ENV_FILE" open "$ENV_FILE" echo "Created $ENV_FILE with placeholders. Fill it in, save, then re-run the skill." >&2 exit 0 fi # Stage a /tmp mirror the simulator process can read; trap deletes it at run-end. cp "$ENV_FILE" "$STAGED_ENV" chmod 600 "$STAGED_ENV" ``` ## Scenario fixture preflight Before writing the temporary Swift test file, verify that the selected scenarios are valid against the live store. The smoke suite should fail when the assistant regresses, not when a demo-store fixture silently disappeared. Use this order: 1. Load only the scenarios being run. 2. Inspect each scenario's optional `fixtures` block first. 3. Infer obvious fixtures from the prompts and `rubric_notes` only when the block is absent. Example: `product called "winter" something; the jacket one` needs at least two searchable products containing `winter`, one of which is clearly a jacket. 4. Verify fixtures through the WooCommerce REST API using `WOO_SITE_URL`, `WOO_USERNAME`, and `WOO_APP_PASSWORD`. 5. Create or update only smoke-owned records. Use stable keys such as SKU, email, or metadata, and prefix them with `woo-ai-smoke-`. 6. Never delete merchant data. Do not mutate non-smoke-owned records just to satisfy a scenario. 7. If a fixed ID in a scenario cannot be guaranteed by the API, report a fixture error instead of treating the run as a model regression. Fixture blocks are intentionally simple JSON embedded in `baseline.json`: ```json "fixtures": { "products": [ { "sku": "woo-ai-smoke-winter-jacket", "name": "Woo AI Smoke Winter Jacket", "type": "simple", "status": "publish", "regular_price": "89.00", "manage_stock": true, "stock_quantity": 7, "stock_status": "instock" } ] } ``` Parse the dotenv file safely. `WOO_APP_PASSWORD` may contain spaces, so do not `source` it in shell unless it is quoted. Use a parser that treats each line as `KEY=value` and preserves the value verbatim: ```python from pathlib import Path def read_store_env(path=Path.home() / ".woo-ai-smoke/store.env"): values = {} for raw in path.read_text().splitlines(): line = raw.strip() if not line or line.startswith("#") or "=" not in line: continue key, value = line.split("=", 1) values[key.strip()] = value.strip().strip('"').strip("'") return values ``` For products, lookup by SKU first: ```bash curl -fsS -u "$WOO_USERNAME:$WOO_APP_PASSWORD" \ "$WOO_SITE_URL/wp-json/wc/v3/products?sku=woo-ai-smoke-winter-jacket" ``` If the product is missing, create it with `POST /wp-json/wc/v3/products`. If it exists and is smoke-owned by SKU, patch it with the fixture values. Leave fixture products published so future smoke runs reuse them. ## Swift smoke template Write to `Modules/Tests/WooAIAssistantTests/SmokeRespondContractTests.swift`. **Always this path — the skill discards it at the end.** ```swift import Foundation import Testing @testable import WooAIAssistant struct SmokeRun { struct Scenario { let id: String let category: String let turns: [Turn] } struct Turn { let prompt: String let autoDeclineWrites: Bool } static let samplesPerScenario = SAMPLES_PLACEHOLDER // 1 for ad-hoc, 3 for default static let scenarios: [Scenario] = [ // FILLED IN by the skill from baseline.json or the ad-hoc args. // Each scenario expanded to `samplesPerScenario` arguments to // @Test so Swift Testing parallel-runs them. ] // Expand each scenario N times so swift-testing runs N independent // samples per scenario in parallel. static let expanded: [(Scenario, Int)] = scenarios.flatMap { s in (1...samplesPerScenario).map { (s, $0) } } @Test(arguments: expanded) func runScenario(_ arg: (scenario: Scenario, sample: Int)) async throws { guard let creds = WooAssistantHeadless.credentialsFromStoreEnv() else { return } let harness = WooAssistantHeadless(credentials: creds) for (index, turn) in arg.scenario.turns.enumerated() { let turnNum = index + 1 // `resolveConfirmation` returns a `ConfirmationDecision` for each // pending confirmation. `.decline` blocks the write. If // `autoDeclineWrites` is true we must return `.decline`. Getting // this inverted means the demo store actually mutates // (destructive writes get approved). let resolver: WooAssistantHeadless.ConfirmationResolver = { _ in turn.autoDeclineWrites ? .decline : .approve } let result: WooAssistantHeadless.ConversationTurnResult do { result = try await harness.send(turn.prompt, resolveConfirmation: resolver) } catch { print("[smoke|#\(arg.scenario.id)|\(arg.scenario.category)|s\(arg.sample)|t\(turnNum)] THREW: \(error.localizedDescription)") return } Self.dump(scenario: arg.scenario, sample: arg.sample, turn: turnNum, prompt: turn.prompt, result: result) } } static func dump(scenario: Scenario, sample: Int, turn: Int, prompt: String, result: WooAssistantHeadless.ConversationTurnResult) { let tools = result.toolCalls.map(\.name) let toolArgs = result.toolCalls.map { "\($0.name)(\($0.argumentsJSON.prefix(120)))" } let cards = Array(Set(result.cards.map(\.kind))).sorted().joined(separator: ",") let confirmations = result.confirmations.map { "\($0.toolName)[\($0.classification)]=\($0.decision)" } let fail = result.failureMessage ?? "" let textEscaped = result.assistantText .replacingOccurrences(of: "\n", with: "\\n") .replacingOccurrences(of: "\"", with: "\\\"") print("[smoke|#\(scenario.id)|\(scenario.category)|s\(sample)|t\(turn)] prompt=\"\(prompt)\" n=\(tools.count) tools=\(tools) toolArgs=\(toolArgs) cards=[\(cards)] confirmations=\(confirmations) fail=\"\(fail)\" text=\"\(textEscaped)\"") } } ``` ## Running ```bash xcodebuild -workspace WooCommerce.xcworkspace \ -scheme WooAIAssistant \ -destination 'platform=iOS Simulator,name=iPhone 17' \