smoke-testing

name	smoke-testing
description	Use when Product Owner defines smoke prompts or deploy handoffs, and when AI Developer / Deploy phase executes post-deploy smoke. Do NOT use for pre-deploy unit/integration/source gates alone; smoke means proving the operator-visible outcome on the deployed surface with ground-truth diagnostics.
source_inspiration	conceptual-only
license_status	clean-room-original
fusebase_flow_version	3.1
risk_level	high
invocation	automatic
expected_outputs	["outcome-based smoke prompts in docs/specs/<slug>/verification-gate.md","deploy-handoff smoke contract in docs/tmp/handoff/<date>-<slug>-deploy.md","smoke evidence in docs/tmp/handoff/<date>-<slug>-smoke/","smoke result table in deploy report"]
related_workflows	["smoke-verification.md","greenlight-deploy.md","live-user-verification.md"]
hook_dependencies	["none"]

Smoke Testing

Style: Mode-B-lite. Outcome-first smoke discipline for Product Owner definition and AI Developer execution.

Purpose

Prevent pre-outcome signals from being mislabeled as smoke success. A smoke test passes only when the operator-visible action works on the deployed surface and the system's ground-truth diagnostics show no hidden failure.

When to invoke

• Product Owner drafts verification-gate.md for a user-facing or operator-facing change.
• Product Owner drafts docs/tmp/handoff/<date>-<slug>-deploy.md and includes S1..Sn smoke prompts.
• AI Developer / Deploy phase runs post-deploy smoke prompts.
• A previous smoke missed a production bug and the smoke method must be corrected.
• Operator asks whether smoke evidence is sufficient.

Do not invoke when

• The task is only pre-deploy source validation: lint, typecheck, unit tests, static bundle checks, file hash checks.
• No deployed or runtime behavior exists to exercise.
• A live user session is required but consent/session handling is not available; invoke workflows/live-user-verification.md first.

Required inputs

Input	Where it lives	If missing
Operator-visible success criterion	verification-gate.md smoke section	PO must write it before designing checks
Smoke prompts S1..Sn	docs/specs/<slug>/verification-gate.md	Stop; smoke cannot be improvised after deploy
Deployed surface / base URL / command	deploy handoff	Stop; ask PO to amend handoff
Ground-truth diagnostic surface	smoke prompt field: logs, request dumps, DB row, rendered UI, response body	Stop; PO must define it or explicitly mark unknown
Existing smoke harness	docs/specs/<slug>/smoke.spec.ts, script path, curl command, or manual steps	Use if present; do not substitute weaker checks silently
UI / browser plan, if applicable	S fields or templates/smoke-test-playwright.md	Require route, viewport, primary action, stable locators, auth plan, unique test data, expected outcome, diagnostics
CLI edition map, for Fusebase Apps work	docs/fusebase-cli-edition.md	Continue only if smoke can name another ground-truth diagnostic surface
Evidence directory	docs/tmp/handoff/<date>-<slug>-smoke/	Create before execution

Procedure

Product Owner: define the smoke contract

1. Write the success-criterion sentence first. It must describe the operator-visible outcome, not an implementation signal.
   • Bad: spawn exits 0
   • Good: operator chat reply renders TemplateListCard with the 8 real templates
2. For ACs citing quality patterns (QP-xx), start from the pattern's Verify line in flow-skills/app-quality-patterns/references/ — it is a copy-ready S recipe; adapt routes/selectors/data. Then identify the ground-truth diagnostic surface for each S. Examples: request dump JSON, app error log, server log, rendered DOM, response body, database row, job trace.
3. For Fusebase Apps smoke, use docs/fusebase-cli-edition.md to pick supporting CLI diagnostics such as remote-logs, dev-debug-logs, fusebase-cli, fusebase-dashboards, or fusebase-gate.
4. Separate supporting checks from smoke checks. File hashes, symbol presence, service active, exit code 0, and HTTP auth sanity are supporting checks only; they cannot satisfy smoke by themselves.
5. For UI smoke, include at least one real primary interaction when the ticket creates or changes one: navigate, submit, save, send, search, filter, authenticate, or complete the main workflow. A screenshot-only check is visual evidence, not sufficient smoke for an interactive feature.
6. For UI smoke, name stable selectors or accessible locators for interactive controls and meaningful dynamic output. Prefer selectors tied to purpose/state, not styling or layout.
7. For browser-driven smoke, define route/navigation, viewport coverage, auth/session plan, setup data, unique test values, and cleanup responsibility. Do not rely on existing shared data counts or empty states unless the smoke prepares that state.
8. For UI actions that cross backend boundaries, name both evidence surfaces: browser-visible result and backend/log/API diagnostic. Browser success alone is insufficient if server logs show errors.
9. Add at least one adversarial/falsification probe: "what would still be broken if the static checks passed?"
10. Define S1..Sn with these fields: scenario, route/navigation, steps, expected operator-visible outcome, pass criterion, ground-truth diagnostic to inspect, stable selectors/locators if UI-facing, auth/test data plan, viewport if browser-facing, adversarial check, evidence required, insufficient substitutes.
11. If the real end-to-end action needs operator credentials or a live session, use workflows/live-user-verification.md. If that is not available now, mark the deploy state SHIPPED-pending-operator-smoke, leave spec DRAFT, and provide the exact operator smoke prompt.
12. When drafting deploy handoff, copy the smoke contract verbatim. Do not compress S1..Sn to "run smoke" or "verify feature works."

AI Developer / Deploy phase: execute the smoke

1. Read every S before starting. Restate the success criterion and ground-truth diagnostic in the deploy log/report.
2. For Fusebase Apps smoke, load the supporting CLI diagnostics named by docs/fusebase-cli-edition.md or the smoke contract. Use them to inspect logs, deployed behavior, and app quality evidence after the operator-visible action runs.
3. Use the named harness if one exists. If a script was created for live smoke, run that script or explain why it is impossible; do not replace it with static checks.
4. Execute the operator-visible action. Capture outcome evidence: screenshot, response excerpt, rendered card text, produced artifact, or job result.
5. For browser smoke, run one user flow at a time, preferably in a fresh browser context. Use unique test data and record any created IDs/names for cleanup.
6. Inspect the ground-truth diagnostic after the action, including any delayed dump/log written after process exit. Use a wait window when the system writes diagnostics asynchronously.
7. For UI/backend flows, inspect browser console/network evidence and the backend/log/API diagnostic named in S. Treat either surface showing an error as smoke FAIL.
8. Treat exit code 0, service active, hash match, deployed symbol, and non-401 auth response as necessary-but-insufficient preconditions.
9. For errors shaped like 'X' object is not Y, NoneType, undefined is not, or equivalent type/iteration failures, get the full traceback first. Do not bisect from guesses.
10. If smoke fails, list at least three candidate causes before selecting the next probe. Design a probe that distinguishes them.
11. If end-to-end execution is not feasible, report PENDING-OPERATOR-SMOKE with the missing prerequisite and exact operator steps. Do not claim PASS, DONE, or SHIPPED.
12. Persist evidence under docs/tmp/handoff/<date>-<slug>-smoke/ and include exact paths in the deploy report.
13. Clean up or document test data and external-service side effects before DONE. Keep evidence files; remove throwaway data not needed for audit.
14. If a smoke missed a bug, update future smoke methodology immediately: add the missed outcome or diagnostic surface to the next gate/handoff.

Flaky-result semantics (FAIL-then-PASS is not PASS)

A smoke S that FAILs and then PASSes on retry is FLAKY, not PASS. Retrying until green launders a real defect (race, ordering dependency, state leak, async timing) into a false-green ship.

• Record every attempt in the evidence dir: S<n> attempt 1: FAIL <observed>; attempt 2: PASS — never overwrite or omit the first result.
• FLAKY resolves only when the first failure is reproduced-or-explained:
  • Reproduced: same conditions re-trigger the failure → real bug. Route to flow-skills/validation-and-qa/SKILL.md Sub-mode C (deterministic-system branch); the S verdict is FAIL.
  • Explained: a named, evidenced external cause (deploy-propagation delay confirmed in the platform log; cold-start timeout visible in remote-logs) with the ground-truth diagnostic ATTACHED under docs/tmp/handoff/<date>-<slug>-smoke/; then 3/3 clean under the same conditions after the cause clears before any PASS (matches validation-and-qa Sub-mode C Explained-external). "Probably a network blip" with no diagnostic is not an explanation.
• Until one of those holds, report S<n>: FLAKY (unresolved) — it counts as FAIL against the smoke threshold and blocks DONE like any FAIL.
• Reproduction mechanics and the deterministic-vs-model-variance verdict split live in flow-skills/validation-and-qa/SKILL.md Sub-mode C.

Verification cost discipline (record-then-read default)

Evidence rules above say WHAT counts; this says HOW to obtain it economically. Agent-side watching (re-reading state every cycle while the system works) costs tokens linear with wall-clock regardless of how little happens — measured at ~10× the cost of reading the same evidence once afterward.

• Default = record-then-read: let the system run unobserved and read its durable evidence surfaces ONCE after the run — journals, run records, logs, event trails, snapshots, outcome docs. Prefer the system's own records over agent-side polling.
• If the system under test has no durable evidence surface, that absence is itself a finding (observability gap) — report it; don't compensate by watching.
• The only exception: the first live drive of freshly-changed code hunting UNKNOWN failure modes, where catching the defect at the moment it happens is the point. Bound it (one flow, one watch window), then revert to record-then-read.
• A long-running verification plan states which mode it uses and why (one line in the S or gate plan).
• Delegated sessions: combine with the turn-completion rule (task-delegation) — record-then-read is how a delegated verifier finishes in one turn.

Output artifacts

Artifact	Path or location	Mode
Smoke contract	docs/specs/<slug>/verification-gate.md	Mode B
Deploy smoke instructions	docs/tmp/handoff/<date>-<slug>-deploy.md	Mode B
Evidence files	docs/tmp/handoff/<date>-<slug>-smoke/S<n>-*.{md,log,png,json}	Mode B
Smoke result matrix	templates/deploy-report.md section 4	Mode B
Pending operator smoke prompt	deploy report + PO chat relay block	Mode A + Mode B

Failure cases

Failure mode	Detection	Response
Pre-outcome signal claimed as smoke	evidence is only exit code/hash/service/curl auth	Mark smoke incomplete; require operator-visible outcome + ground-truth diagnostic
Ground-truth diagnostic missing	S has no log/dump/artifact/DOM/row to inspect	PO amends smoke contract before deploy
Harness exists but was skipped	deploy report uses weaker manual/static check	Mark smoke incomplete; run harness or report pending operator smoke
UI smoke only checks render/screenshot	primary interaction not exercised	Mark incomplete; run the real interaction or document pending operator smoke
UI smoke relies on brittle selectors	selectors are CSS/layout/text fragments likely to drift	Add stable purpose/state selectors or accessible locators before claiming PASS
Browser smoke plan is incomplete	missing route, viewport, auth plan, unique data, expected outcome, or cleanup	Mark incomplete; amend S before execution
Shared-state assumption	test expects exact counts or empty state from data it did not create	Create unique data or isolate state; otherwise smoke cannot prove the outcome
Browser and backend disagree	UI appears correct but console/network/server diagnostics show errors	Smoke FAIL; attach both evidence surfaces
End-to-end unavailable	missing live cookie/session/API credential	Use live-user workflow or mark PENDING-OPERATOR-SMOKE; spec stays DRAFT
Smoke fail after deploy	S observed != expected or diagnostic shows error	Do not mark DONE; surface rollback vs fix-forward options
Hidden delayed failure	process exits 0 but later dump/log contains exception	Smoke FAIL; attach dump/log evidence
Flaky smoke (FAIL then PASS on retry)	first attempt failed, retry passed, no reproduction or evidenced explanation attached	Verdict FLAKY (unresolved) = FAIL against threshold; record all attempts; route first failure to validation-and-qa Sub-mode C

Escalation path

• Missing smoke contract -> Product Owner amends verification-gate.md.
• Live account required -> follow workflows/live-user-verification.md.
• Smoke threshold not met -> operator chooses rollback or fix-forward in chat text (FR-19).
• Repeated escaped smoke bug -> add/update docs/problem-catalog/<slug>/problem.md and amend this skill or project smoke references.

Anti-patterns

• Do not call static deployment checks "smoke."
• Do not declare PASS from exit code 0 alone.
• Do not re-run a failed S until it passes and report PASS; FAIL-then-PASS is FLAKY until the first failure is reproduced or explained with attached diagnostics (§ Flaky-result semantics).
• Do not declare PASS from file hashes, symbol presence, service active, tick latency, or auth transport alone.
• Do not substitute an easier probe for the sufficient end-to-end probe without marking the result partial.
• Do not use screenshots alone as PASS evidence for an interactive UI change.
• Do not rely on brittle layout/style selectors when stable purpose/state selectors or accessible locators can be provided.
• Do not rely on exact shared-data counts, preexisting records, or empty states unless the smoke creates or isolates the data.
• Do not run browser automation for API-only, unit-only, load, or performance checks when a lighter deterministic probe proves the same outcome.
• Do not trigger real customer-visible external-service side effects without an approval/sandbox plan.
• Do not mark spec DONE when smoke is failed or pending operator execution.
• Do not hide missing credentials behind a green status.

Clean-room note

Original Fusebase Flow content. Derived from operator-provided retrospective patterns and generalized into repo-local workflow discipline; no third-party code, prompts, skill files, or hook scripts are copied. See docs/source-map.md.