acceptance-test-authoring

name

acceptance-test-authoring

description

How to set up and write acceptance tests for this project — specs are Markdown (spec.md) with fenced Gherkin, extracted to .feature files at test time; Cucumber runner configuration (extraction, effective-spec composition of source-of-truth and active delta specs, archive exclusion, HTML report); gherkin-lint over the extracted output; Page Object Model conventions for step definitions. Use when creating or modifying anything under acceptance-tests/, configuring the cucumber runner, writing or refactoring step definitions, linting specs, or implementing tasks from an OpenSpec change that involve acceptance tests.

Acceptance Test Authoring

The acceptance suite executes the Gherkin specs that live under openspec/ against the running application. Specs are Markdown files (spec.md) holding prose plus classic Gherkin inside fenced code blocks; the runner extracts the Gherkin into real .feature files on every run. Three sets of rules govern the suite: spec format and extraction (how Gherkin gets out of the Markdown), runner invariants (what the suite runs and how), and code organization (Page Object Model). The machinery is implemented by the reference files, described first.

Reference files

Copy all four files into acceptance-tests/ — together they are the canonical runner. Destination filenames are load-bearing:

• references/extract-gherkin.cjs — the Gherkin extraction. Exports extractAll() (wipe + rebuild .extracted/ from every spec.md under openspec/, archive excluded) and a CLI form (node extract-gherkin.cjs [openspecDir] [outDir]) so specs can be linted before an acceptance-tests/ project exists. Destination name MUST stay extract-gherkin.cjs — cucumber.cjs requires it as ./extract-gherkin.cjs.
• references/cucumber.cjs — the runner configuration. Runs extractAll() synchronously at config load (so plain cucumber-js always sees a fresh extraction), wires effectivePaths() into the default profile (so npm test runs the effective spec) and defines the specs profile (source-of-truth-only regression run, npm run test:specs). Destination name MUST stay cucumber.cjs — cucumber-js discovers its configuration by that exact name.
• references/openspec-effective-paths.cjs — the effective-spec extraction. Exports effectivePaths(), which computes the suite composition over the .extracted/ tree (source of truth + active deltas, superseded rules excluded — see Effective-spec extraction) and prints the composition report to stderr. Destination name MUST stay openspec-effective-paths.cjs.
• references/gherkin-lintrc.json — the pinned lint config. Copy to acceptance-tests/.gherkin-lintrc (gherkin-lint auto-discovers that name and has no built-in defaults). The off-rules are load-bearing — see Linting specs.

devDependencies: @cucumber/cucumber, glob, cheerio, gherkin-lint.

Spec format and extraction

A spec is openspec/specs/<capability>/spec.md (source of truth) or openspec/changes/<id>/specs/<capability>/spec.md (delta). Markdown prose — headings, rationale, links — may appear anywhere; only ```gherkin fences are executable:

• Fences open with ```gherkin at column 0 (3+ backticks, info string exactly gherkin) and close with at least as many backticks at column 0.
• A file may hold multiple gherkin fences; concatenated in file order they must form exactly one Feature: document. Gherkin ignores blank lines, so prose gaps between fences are harmless.
• Tags stay in the same fence as (directly above) the Feature: line.
• # @openspec: delta markers are ordinary gherkin comments inside a fence, immediately above their Rule:.

extractAll() writes each spec.md to acceptance-tests/.extracted/<same-relative-path>/spec.feature. Line fidelity is the core invariant: lines inside gherkin fences are copied verbatim at their original positions; every other line (prose, fence markers, non-gherkin fence bodies) becomes an empty line, so the extracted file has the IDENTICAL line count and line N of the .feature is line N of the .md. gherkin-lint messages, cucumber failure locations and effective-spec line targeting all point at valid spec.md lines with zero translation. Never "improve" the extractor to collapse blank lines.

Path mapping: cucumber's own output and the HTML report show extracted paths — read .extracted/X/spec.feature:N as openspec/X/spec.md:N (same line number, always).

.extracted/ is gitignored, wiped and rebuilt on every run, and never edited by hand. The wipe is an invariant, not an optimization — a stale extraction would keep deleted or renamed capabilities executing.

Extraction edge cases (all deliberate — silent drops are the failure mode to fear):

Case	Behavior
Unclosed fence	Error with file:line of the opener
Zero gherkin fences in a spec.md	Error — a spec must contain gherkin
Indented ```gherkin opener	Hard error — silently ignoring it would silently drop scenarios
```gherkin extra-text	Not a gherkin opener (info string must be exactly gherkin) — treated as an ordinary fence, contents blanked
Non-gherkin fences (```js, plain ```, 4+ backticks)	Tracked, contents blanked — a ```gherkin quoted inside a longer documentation fence cannot false-trigger
Gherkin docstrings delimited by ```	Safe — docstrings are indented; fence closers require column 0
Files other than spec.md	Ignored — discovery globs are anchored to spec.md (proposal/design/tasks excluded structurally)
Legacy .feature files under openspec/	Never run — extraction prints a WARNING naming them

Runner invariants

1. acceptance-tests/ is an independent Node project at the repo root (own package.json). Its hooks boot the application before the suite and shut it down after — the suite must be runnable with a single npm test. Scripts: test, test:specs, lint:specs. The repo .gitignore covers acceptance-tests/.extracted/ and acceptance-tests/reports/.

2. The default run executes the effective spec: every openspec/specs/**/spec.md (source of truth) with every active change's delta (openspec/changes/*/specs/**/spec.md) applied — i.e. exactly what the source of truth will become once those changes are synced/archived — extracted to .extracted/ and composed there. Composition, per Rule: in an active delta:

   Rule in an active delta	Source-of-truth version	Delta version
   (untouched)	runs	—
   # @openspec: ADDED	—	runs
   # @openspec: MODIFIED	not discovered	runs
   # @openspec: REMOVED	not discovered	(no scenarios)
   # @openspec: RENAMED	runs (name change only)	(no scenarios)

3. Superseded rules (MODIFIED/REMOVED by an active delta) are excluded at discovery time via cucumber line-targeted paths (spec.feature:27:33 runs only the scenarios starting at those lines — valid spec.md lines too, by line fidelity). Never edit or tag spec files to skip them (the source of truth stays pristine until sync) and never mark them runtime-skipped (skipped counts pollute the zero-pending completion signal — a superseded rule is replaced, not unfinished). Every exclusion is announced via the composition report on stderr (see Effective-spec extraction, step 6) — an excluded scenario must never be silently absent from results or the HTML report.

4. A green effective suite is the gate for sync/archive — and sync/archive must never change suite results. At propose time the suite goes red on exactly the delta's new/changed scenarios (that red set is the implementation work list); at completion it is fully green; syncing and archiving then only collapse the composition back to the source of truth. Never sync or archive on red.

5. Specs under openspec/changes/archive/ must NEVER execute. Archived changes are historical deltas already merged into openspec/specs/; running them re-executes stale duplicates. This is non-negotiable. The extractor already skips the archive (nothing under changes/archive/ reaches .extracted/), and the path composition filters it again — defense in depth.

6. Never use negated (!) globs to exclude the archive — cucumber-js does not support negation in paths and silently ignores such patterns (the exclusion looks configured but does nothing). Instead:

   • use include-only patterns (changes/*/specs/** structurally cannot match changes/archive/<name>/specs/** — archive adds a path level), and
   • resolve file lists explicitly with globSync and apply a defensive changes/archive/ filter.

   Use the four reference files as the canonical runner config — copy them as-is for new projects.

7. Alongside the default run, provide a specs profile (npm run test:specs → cucumber-js -p specs) that executes the source of truth only — an as-is regression run (over .extracted/specs/, freshly extracted by the same config load).

8. Every test run generates an HTML report (reports/cucumber-report.html) via the cucumber html formatter.

9. Verify the composition whenever the runner config, the extractor, or the openspec/ tree changes: run npx cucumber-js --dry-run and confirm .extracted/ was rebuilt, no loaded scenario originates from openspec/changes/archive/, no scenario appears twice, and no scenario of a superseded source-of-truth rule is loaded. Quick check of resolved paths (must show only .extracted/ entries, some line-targeted):

   node -e "console.log(require('./cucumber.cjs').default.paths)"
   

Effective-spec extraction

The extraction is defined twice — use the form that fits the stack:

• As a function: effectivePaths() in references/openspec-effective-paths.cjs — the canonical, executable definition for cucumber-js projects.

• As a procedure, for any other runner:

  1. Extract — run the Gherkin extraction: every spec.md under openspec/specs/ and openspec/changes/*/specs/ (archive excluded) becomes .extracted/<same-path>/spec.feature with identical line numbers.

  2. Collect active deltas — every .extracted/changes/*/specs/**/*.feature, with a defensive changes/archive/ filter.

  3. Extract superseded rules — scan each delta for # @openspec: <OP> marker comments bound to the next Rule: line (use the regexes from the schema's format: block); record (capability, rule name) for MODIFIED and REMOVED. The capability is the path segment after the delta's last specs/. ADDED and RENAMED supersede nothing. Fail if two active changes supersede the same rule; warn if a superseded rule doesn't exist in the source of truth (delta drift).

  4. Filter the source of truth — for each .extracted/specs/<capability>/spec.feature containing superseded rules, select only the scenarios of non-superseded rules (in cucumber: line-targeted paths spec.feature:L1:L2); omit the file entirely if nothing survives; include untouched files whole.

  5. Add every delta file whole.

  6. Print the composition report to stderr — each superseded rule, the change that superseded it, and every left-out scenario with its source spec.md file:line (identical line numbers, by line fidelity), plus a summary count, so an excluded scenario is never silently absent from results or reports:

     [effective-spec] user-signup / Rule: A signup SHALL require an email address and a password
     [effective-spec]   superseded by change: signup-email-verification
     [effective-spec]   left out: Signing up with valid details (../openspec/specs/user-signup/spec.md:12)
     [effective-spec]   left out: Rejecting a signup with no email (../openspec/specs/user-signup/spec.md:37)
     [effective-spec] 2 source-of-truth scenario(s) excluded; delta versions run from openspec/changes/
     

Linting specs

npm run lint:specs = extract, then lint the extracted output:

"lint:specs": "node extract-gherkin.cjs && gherkin-lint .extracted"

• Pass .extracted as a directory argument — a quoted '**' glob silently matches nothing through the dot-directory.
• gherkin-lint has no default rules; it requires .gherkin-lintrc (copy references/gherkin-lintrc.json to that name). The pinned off-rules are load-bearing: whitespace rules (no-multiple-empty-lines, new-line-at-eof, indentation) would flood on the extraction's blank-line padding, and the dupe-name rules fire on legitimate SOT/delta pairs (same Feature name; MODIFIED deltas copy scenario names). Without them, "fix all lint issues" is unsatisfiable.
• Reported line numbers are valid in the source spec.md files.
• Known limitation: gherkin-lint's AST rules do not descend into Rule: children, so e.g. no-unnamed-scenarios misses scenarios nested under a Rule. Line-based rules (trailing spaces, tags) and feature-level rules work fine.

Linting a spec before an acceptance-tests/ project exists (e.g. from a specs-authoring session): run the extractor from the skill references — node .claude/skills/acceptance-test-authoring/references/extract-gherkin.cjs openspec acceptance-tests/.extracted && npx gherkin-lint --config .claude/skills/acceptance-test-authoring/references/gherkin-lintrc.json acceptance-tests/.extracted (the output dir is gitignored; writing it via Bash does not violate the spec/code zone split, which guards file edits only).

Page Object Model

Step definitions must read as intent; all knowledge of the UI lives in page objects.

• Page objects live in acceptance-tests/support/pages/, one per screen or flow (e.g. signup-page.js, login-page.js, dashboard-page.js).

• A page object encapsulates all UI identifiers: routes/URLs, form field names, CSS selectors and element ids. Parse responses with cheerio — never with regexes over raw HTML.

• Page objects expose intent-level methods, e.g. open(), submitSignup({ email, password }), errorMessage(), confirmationLink(). They receive the World (or its HTTP client) and use it for requests.

• Step definitions contain no selectors, no regexes, no URLs — only page-object calls and assertions:

  When('they submit a valid email and password', async function () {
    this.result = await this.signupPage.submitSignup({
      email: '[e-mail]',
      password: 'correct-horse-battery-staple',
    });
  });
  
  Then('an error message is shown', function () {
    assert.ok(this.signupPage.errorMessage());
  });
  

• The World stays a thin HTTP client (request/response state, redirect helper). It holds page-object instances; it does not parse pages itself.

• When the UI changes an identifier, exactly one page object should need editing.

Workflow cadence

Implement one pending step definition at a time: run the suite so the step fails for the right reason (red), implement until it passes (green), then commit. The effective suite's red scenarios at propose time are exactly the change's work list. Finish only when every scenario passes with zero pending/undefined steps and the HTML report is generated — then, and only then, sync and archive.