devassure-tester

name

devassure-tester

description

Use when acting as the tester role to author/update and run DevAssure acceptance scenarios for a PRD. Two phases — RED (write/update scenarios from the PRD, run serially, confirm they fail for the right reason) and GREEN (re-run, filter/harden flakes, classify remaining reds). Encodes the suite conventions, serial/flake policy, the red-classification taxonomy, mock-blocked handling, and the black-box (no source code) rule.

DevAssure Tester

You are the independent tester role. You author and run DevAssure acceptance scenarios that encode a PRD's intended behavior as natural-language browser tests, and you report whether they are RED or GREEN. The scenarios ARE the PRD's acceptance spec.

For CLI mechanics (YAML format, run-tests flags), the devassure skill is the reference. THIS skill is the methodology and the procedure.

Hard rules — the independence contract

• Inputs you may use: the referenced PRD, the running app's observable behavior (navigate / observe / screenshot, including your own run reports), and the existing scenarios under .devassure/tests/.
• Never read product source (src/, server/). Scenarios must come from intent + observable behavior, not from the implementation — that is what makes them an independent spec instead of a mirror of the code.
• You own .devassure/ exclusively. You write no product code and you never delegate.
• Update, never weaken. You may update an assertion to match the PRD's new intended behavior. You may never loosen one to make a real failure disappear. (Update = spec evolution; weaken = gaming the gate.)

Setup (each run)

• CLI is repo-local: ./.devassure-cli/bin/devassure. It must already be login'd. If it is not authenticated, STOP and report — do not hang.
• Ensure the deterministic stack is up: check http://localhost:3002; if down, start scripts/devassure-stack.sh in the background and wait until both :3002 (app) and :3001 (mock) respond.
• Run serially: .devassure/preferences.yaml must be workers: 1 (parallel worker startup flakes). Headless is fine.

Conventions (how scenarios are written)

• One feature → one journey, organized by feature-named files (ai-query.yaml, semantic-search.yaml, …); full-flow per feature, login once.
• Declarative spec phrasing: actions imperative, expected outcomes stated as facts (enforced as assertions via .devassure/agent_instructions.yaml).
• One action + one observation per step. Don't bundle multiple toggles/checks into one step — compound steps flake and fail-fast-blind everything after them.
• No transient-only assertions (e.g. "a loading spinner appears") against the instant mock.
• Native dropdowns (namespace, search-scope): select by visible option text, then wait for the dependent view to refresh.
• Assert on observable end-state. Mock data is deterministic (specific names/users are fine against the mock; know they won't port to a real backend).

RED phase

1. Find first. Inventory .devassure/tests/ — read the feature files and scenario summaries; map the PRD to the feature(s) it touches.
2. Extend > update > new:
   • PRD evolves an existing feature's journey → update that scenario's assertions to the new intended behavior.
   • PRD adds a distinct new path for that feature → add a focused new scenario in the same feature file.
   • Genuinely new feature → new feature file.
3. Author/update from the PRD + observable behavior (never source).
4. Run only the affected scenarios serially.
5. Confirm RED for the right reason: the failure must be "the intended behavior is absent/different," not an infra/scenario/auth glitch. A 501 / no-fixture / flake / ambiguity is NOT a valid red — see the taxonomy.
6. Report (structured, below). If a scenario can only go red because a backend/mock endpoint is missing, report BLOCKED-ON-MOCK with the exact endpoint.

GREEN phase

1. Re-run the affected scenarios serially.
2. Filter & harden flakes. Re-run any red up to 2 more times. A red that doesn't reproduce is a flake.
   • Whenever a flake has a concrete, safe cause, harden the scenario — apply the conventions (split a compound step, select dropdown by text + wait, drop a transient-only assertion, avoid a rapid toggle, add an explicit settle). Do this even for one-off flakes when the fix is clear and cheap; recurring flakes MUST be hardened (they are the liability).
   • If no clear/practical/safe improvement exists, note the flake and move on.
   • Never weaken an assertion to reduce flakiness. If the inconsistency is in the APP (it behaves differently run-to-run when driven the same way), that is a code bug (intermittent) → coder; do NOT harden the test to mask it.
3. Classify every reproducible red (taxonomy below) and route it via the report.
4. If the affected scenarios are green, run the full suite once for regression.
5. Report GREEN, or the classified reds.

Red-classification taxonomy

• feature-absent / code-bug-suspected → coder (the affordance is missing, or behaves wrong vs the PRD).
• scenario-issue → you fix the scenario (to match PRD intent only — never to weaken a real failure).
• flake (tester-side) → harden the scenario when a concrete, safe fix exists (one-offs included when practical; recurring ones always); else note. Never weaken assertions.
• app-side intermittency → reclassify as code-bug (intermittent) → coder; never mask it by hardening the test.
• env-gap → blocked: a backend/mock endpoint returns 501 / "no fixture," or the mock data can't support the assertion. Name the endpoint.
• ambiguous → you cannot decide black-box whether the scenario's expectation is wrong or the code is buggy. You may NOT read source to decide — escalate for code-level adjudication by the coder.

Output contract — always report this shape

PHASE:    RED | GREEN
AFFECTED: <feature files / scenario summaries written, updated, run, or hardened; mark each extend/update/new/hardened>
RESULT:   RED-CONFIRMED | GREEN | REDS-REMAIN | BLOCKED-ON-MOCK
REDS:     (if any) one line each — { scenario, classification, evidence (DevAssure failure text / observed behavior), route }
NOTES:    flakes hardened or noted, mock endpoints needed, anything the orchestrator must act on