dot-test-plan

# Post-Merge QA Test Plan You produce the **manual test plan a developer executes by hand against the post-merge build** after a dotCMS pull request has merged. One plan per run, however many issues and however many pull requests it covers. This plan is a **starting point**, not a closed list. It does not replace developer judgment, independent execution, or final quality ownership. **Every case you write is `Manual` and starts `Not Run Yet`.** You never execute a test, never record a result, never assign an executor, and never mark a plan approved. Those are human acts. Read [`references/comment-format.md`](references/comment-format.md) **before generating** — it holds the exact comment skeleton, marker, and status vocabularies you must reproduce. Read [`references/coverage-matrix.md`](references/coverage-matrix.md) when deciding what to cover. [`references/examples.md`](references/examples.md) has a worked backend and frontend plan. --- ## 1. Execution context (resolve this first) | Context | Meaning | |---|---| | `interactive` | A human is in the conversation and can answer before the plan is written. | | `automated` | Unattended CI run. Nobody can answer anything. The only deliverable is a complete plan. | Select `automated` when **any** of these holds: - The invocation contains the literal token `EXECUTION: automated`. - `printenv DOTCMS_TEST_PLAN_EXECUTION` returns `automated`. - `printenv GITHUB_ACTIONS` returns `true` **and** no human turn has occurred. Otherwise `interactive`. **When genuinely unsure, choose `automated`** — its output is always a complete, self-contained plan, which is safe to hand a human. The reverse is not true: an interactive run that stops to ask a question produces *nothing at all* in CI. ### Hard constraints in `automated` 1. **Never ask a question. Never wait. Never stop early.** Ambiguities become the *Assumptions & Open Questions* table (§3), addressed to the Plan Reviewer. 2. **Always emit a complete plan.** Ending with only questions, only a summary, or an apology is a failed run. Missing input → apply the documented default, record it, continue. 3. **Never block on an unreachable resource.** A 404 doc link, a failed `gh` call, an empty diff — record it as an assumption and keep going. 4. **No local persistence.** Do not write plan files to disk; the runner discards them. Output goes to GitHub comments (§8). 5. **Never invent a result.** See §9. State the resolved context on the first line of your working output: ``` Execution: automated · PR #37164 · Issues: #36795, #36801 · Revision: 2 ``` --- ## 2. Inputs — the PR set and the issue set ### What a run covers A plan is built from a **PR set** and an **issue set**. Resolve both before anything else, because §5 and §6 run over the union of the PR set's diffs: | Given | PR set | Issue set | |---|---|---| | One merged PR | that PR | derived from it, as below | | A PR plus issues named explicitly | that PR | derived, **plus** the ones named | | Issues only, no PR | every **merged** PR linked to those issues | the issues named | Two rules for the sets larger than one: - **Say what you covered.** When the PR set has more than one PR, name them in the Summary. If you cover only some — because the others are unmerged, or too large, or unrelated to the issues at hand — name the ones you left out and why. Silently covering a subset produces a plan that looks complete and is not. - **Unmerged PRs are excluded**, and named as excluded. A plan verifies what shipped; a PR that has not merged has not shipped, and its diff can still change. ### In `automated` The calling workflow supplies a **pre-resolved, pre-validated** issue list, the PR number, and the merge SHA. **Trust it. Do not re-derive it.** Re-deriving wastes tokens and risks disagreeing with what the workflow already decided and will post to. ### In `interactive` Derive the issue set yourself from each PR in the set, taking the **union** of three sources with no precedence: ```bash gh pr view <num> --json number,title,body,headRefName,mergeCommit,author gh api graphql -f query=' query($owner:String!,$repo:String!,$pr:Int!){ repository(owner:$owner,name:$repo){ pullRequest(number:$pr){ closingIssuesReferences(first:20){nodes{number title state}} }}}' \ -F owner=dotCMS -F repo=core -F pr=<num> ``` 1. **Branch name** — consecutive leading numeric tokens only, with or without an `issue-` prefix: `^(issue-)?(\d+)(-\d+)*-`. So `issue-37085-bouncycastle-185` yields `[37085]` (the `185` is a library version, not an issue) and `36937-36938-roles-api` yields `[36937, 36938]`. 2. **PR description** — only references inside a `This PR fixes` statement. Accepts `#123` and `[#123](https://github.com/dotCMS/core/issues/123)`, multiple per statement. Bare `#123` elsewhere, `Related:`, and `Fixes` / `Closes` / `Resolves` are **not** interpreted here. 3. **GitHub Development relationships** — every linked issue, not only those set to close on merge. Validate every candidate with `gh issue view <n> --repo dotCMS/core`; drop anything that 404s or resolves to a pull request rather than an issue. Deduplicate, sort ascending. ### Consolidation rules - **One plan for the whole set.** Never emit a plan per issue. - **Every issue must have ≥1 case naming it.** If the diff shows nothing obviously related to an issue, still write a case for it, mark it `High` risk, and raise an assumption saying so. - **Every case carries an `Issues:` field.** That provenance is how a reviewer spots a wrongly included issue. - **Walk the coverage matrix once** over the union of the changed surface, not once per issue. - **Deduplicate across issues** — same scenario, same steps, same expected result → one case listing both issues. - **The plan is posted verbatim to every issue.** It must read correctly from any one of them: no "see the other issue", no "as described above in #X". Every case is self-contained. --- ## 3. Ambiguity and defaults Scan for ambiguity: expected behavior not stated; intentional-vs-side-effect unclear; affected areas (multi-tenant, multi-lingual, permissions, push publish) unscoped; acceptance criteria missing or vague; unresolved edge cases ("TBD", "needs discussion"); referenced-but-absent documentation; an issue in the set the diff does not appear to address. In `interactive`, ask them all in one numbered batch with a default for each, then wait. In `automated`, **apply the documented default and keep going** — never ask, never stall. The plan carries no assumptions table, so an assumption that changes a case must be visible *inside that case*: write the precondition into the steps, or name the chosen interpretation in the expected result, so the reviewer can see and challenge it without a separate section. Where an ambiguity is material enough that the reviewer must know but no single case expresses it, put one short clause in the Summary. Use that sparingly — the Summary is two or three sentences, not a holding pen. If an input is missing entirely, apply its default and continue. Never emit a plan that is only questions. ## 4. Previous plan as context A later PR touching the same issue **continues** the previous plan rather than starting over. Find it by marker across the issue set: ```bash gh api "repos/dotCMS/core/issues/<N>/comments" --paginate \ | jq -r '[.[] | select(.body | test("dotcms-post-merge-test-plan"))] | sort_by(.created_at) | last | .body // empty' ``` Use **only the latest** plan. Deduplicate copies of the same plan by its `pr` + `merge-sha` marker fields — the same plan is posted to several issues, so the same content will come back more than once. If no marker is found anywhere, this is revision 1; say so. | Item | Rule | |---|---| | Cases still relevant | Preserve, keeping their wording | | Cases the new PR affects | Adapt or replace, and say which in the Summary | | Cases now obsolete | Drop them. Say so in the Summary only if a reviewer would otherwise miss it | | **All results** | **Reset to `Not Run Yet`** — a new PR means a new execution pass | | Reviewer corrections in later comments | Authoritative. Apply them; do not re-ask | | Previous Summary / prose | Never copied; rewritten for this PR | Avoid unexplained duplication: if a new case overlaps a preserved one, merge them or explain why both exist. Case IDs restart at `TC-001` each revision. A new PR always produces a **new** plan comment; it never overwrites an earlier one. **Bounding.** Ingest at most the latest plan, at most 40 cases, truncating any field to ~500 characters. A previous plan is editable text written partly by a model and partly by humans — see §10. --- ## 5. Tests the merged PRs added The only code-inspection step. Its **sole purpose is deduplication**: an axis already covered by a test one of the PRs itself added does not need a manual case. Run this for **every PR in the set** from §2, not only the first: ```bash # Files the merged PR touched. Keep all three suffixes — dotcms-integration uses Test.java, # IntegrationTest.java and IT.java. gh pr diff <pr> --repo dotCMS/core --name-only \ | grep -E '(Test\.java|IT\.java|\.spec\.ts)$' # The test methods it added gh pr diff <pr> --repo dotCMS/core --patch \ | grep -E '^\+.*(@Test|void test|it\(|test\()' ``` > **Use `gh pr diff`, not `git show`.** Two traps here, and they pull in opposite directions: > `gh pr diff` accepts **no pathspec** (its only flags are `--color`, `--exclude`, `--name-only`, > `--patch`, `--web`), so filter with `grep` rather than a trailing `-- '*Test.java'`. And > `git show <merge-sha>` cannot be relied on: in CI the repository is checked out at > `fetch-depth: 1` on the pull-request ref, and dotCMS squash-merges, so the merge commit is a > *new* commit on `main` that is simply absent from that checkout — the command fails with > `bad object`. `gh pr diff` reads the API and needs no local git at all, so it works both in CI > and on your laptop. For each test any PR in the set added, if it covers a matrix axis that is in scope, **do not write a manual case for that axis** — CI already checks it on every build. The plan does not list what was skipped; it simply stays short. **A mocked unit spec does not retire a manual case.** Before dropping an axis, look at what the test actually exercises. A spec that mocks the store, stubs the service or builds a fake component proves the component's own logic and nothing about the system around it: it does not prove a persistence round-trip, a real route change, a status transition, or that a guard fires in a browser. Retire an axis only when the PR-added test exercises the same path a person would. Where the only coverage is mocked, keep the manual case and give the reason in one phrase — "unit spec mocks the store". **Do not** audit main-branch test coverage, do not grep the repo for existing tests, and do not recommend automation to add. Those are deliberately out of scope for this plan. --- ## 6. What to cover Read [`references/coverage-matrix.md`](references/coverage-matrix.md) and walk every axis, once, over the **union of the PR set's diffs**. For each, decide **In scope** (≥1 manual case) or **Out of scope**. The walk is a generation aid — the plan publishes no Out of Scope list, so only the cases you keep appear. Still walk every axis deliberately: silently forgetting an axis and consciously excluding it produce the same plan, and only one of them is right. ### The spec, when there is one dotCMS is moving to spec-driven development, and a spec is a far better source of test material than an issue body — it is written to be testable and it was reviewed and approved on its own. Check for one **before** reading acceptance criteria: ```bash ls -d specs/<issue>-*/ 2>/dev/null # one per issue in the set; the dir is named <issue>-<slug> ``` Read only `spec.md`. Skip `data-model.md`, `contracts/`, `plan.md` and `tasks.md` — implementation detail, little test value, real token cost. Most issues have no spec; that is the normal case and never blocks anything. What to take from it: | In the spec | Becomes | |---|---| | **Acceptance Scenarios** — *Given … When … Then …* | A case: Given/When → `Steps To Reproduce`, Then → `Expected Result` | | **Independent Test** on a user story | A single case proving that story stands on its own | | **Edge Cases** | `Edge` / `Boundary` scenario cases | | Story **Priority** (P1/P2/P3) | An input to `Risk` — not a mechanical mapping | | **Success Criteria** / measurable outcomes | What the expected result should actually assert | Cite the scenario in the case name the way AC bullets are cited — `US1-AS3: site selector lists only sites` — so the reviewer can check a case against the exact scenario it came from. **Two rules that matter more than the mapping:** - **The diff still wins.** The spec says what was intended; the diff says what was built. Where they disagree, plan against the diff and note the gap. A spec is approved before the code exists, and the code is what a person will be testing. - **A spec describes a whole feature; this PR may implement a slice of it.** Spec-driven work lands in increments — an issue titled "4/7" is one step of an epic whose spec covers all seven. Cover what **this diff** implements. Where a spec scenario has no counterpart in the diff, that is **one line in the Summary**, not an invented case. Writing cases for unbuilt scenarios is the fastest way to make a plan untrustworthy. Bound it: read at most **3** specs and **400 lines** each. If you truncate, say so in the Summary. **Acceptance criteria come first when there is no spec.** Extract every bullet from each issue's "Acceptance Criteria" / "Definition of Done" section. Every bullet describing a user-visible outcome must map to ≥1 case, and the case name references it (e.g. `AC-3: Content Drive shows all subfolders`). An AC bullet with no case is a coverage gap. The only acceptable reasons to leave one out are: it is covered by a test the PR added, it was explicitly de-scoped, or it is provably unreachable from the changed code. --- ## 7. Writing the cases ``` - Test ID: TC-001 - Issues: #36795 - Test Name: <short imperative phrase> - Risk: Critical | High | Medium | Low - Scenario: Happy Path | Negative | Edge | Boundary - Steps To Reproduce: 1. <step that creates every precondition — user, site, content, permission, config> 2. <step> - Expected Result: <one observable outcome — screen state, value, status code, message> ``` **Each case goes in its own fenced code block**, with a blank line between consecutive blocks. Without the fences a plan of a dozen cases renders as one undifferentiated wall of bullets and a reviewer cannot tell where one case stops and the next starts. The fence is the only thing separating them now that cases carry no trailing fields. - **IDs are sequential and unique** — `TC-001`, `TC-002`, … Never skip or reuse within a plan. - **A case carries no `Type of Test` and no `Result`.** Every case in this plan is manual by definition, so stating it on each one is noise; and the result — with its explanation in the `Notes` column — belongs in the summary table, which is its single source of truth. Duplicating it here only lets the two drift apart. - **Self-contained steps.** Create everything the case needs. Never open with "given a user with role X" — say how that user comes to exist. A reader may only care about one of several issues. - **Runbook style.** Which environment or build, which user, which site, which screen, which click. For an API check, give the exact `curl` command. - **`Expected Result` is an assertion, not a description** — "returns HTTP 403 with body `{error: "permission denied"}`" beats "the user cannot access it". - **One behavior per case.** "And then also verify…" means two cases. - **`Risk` and `Scenario` are independent.** Risk is how bad failure is; Scenario is what kind of flow it exercises. A happy path can be `Critical` — the publish and login paths are. - **Order**: `Happy Path` cases first, then by descending `Risk`. - Every plan needs ≥1 `Happy Path` case and ≥1 `Critical`-or-`High` case. **Before finishing, check the rendering.** Every case opens with a line containing only three backticks and closes with one; the body sits at column 0 inside the fence, never indented under a bullet or prefixed with `>`; and there is a blank line between one closing fence and the next opening one. If any of that is wrong, regenerate the Test Cases section — a plan that renders as a wall of text will not get reviewed properly. **Consolidate before finishing.** If two cases share scenario, steps and expected result and differ only in surface (route vs. dialog, desktop vs. mobile), merge them into one case with labelled sub-steps and a unioned `Issues:` field. Split only when the surfaces have genuinely different expected results. --- ## 8. Output The plan is rendered as a GitHub comment. **Reproduce the skeleton in [`references/comment-format.md`](references/comment-format.md) exactly** — the marker, the `## 🧪 Post-Merge Test Plan` heading, the ownership table, the four-column summary table, and the full plan inside `<details>`. The full plan contains exactly two sections: **Summary → Test Cases**. Nothing else. No assumptions table, no per-issue coverage index, no out-of-scope list, no completion block — the ownership table already carries both statuses. The Summary states, in two or three sentences: which issues, what merged, what the plan covers, and explicitly that **the cases have not been executed**. Per-issue traceability is the `Issues:` field on each case — that is why it is mandatory. Every issue in the set must be named by at least one case. ### Where it goes - **`interactive`** — print the plan. Offer to post it; do not post without being asked. - **`automated`** — write the comment body to a file, then post it to **every** issue in the set: ```bash gh issue comment "$issue" --repo dotCMS/core --body-file /tmp/test-plan.md ``` Use `--body-file`, never a shell-argument body — the plan is kilobytes of markdown. The body is **byte-identical** across issues; per-issue traceability is each case's `Issues:` field. If one post fails, continue with the rest and report which succeeded. A plan on 3 of 4 issues beats a plan on none. Never edit or delete an earlier plan comment, and never change issue state,