pmm-manual-test

Stars13

Forks14

UpdatedJune 10, 2026 at 12:30

Guides PMM manual testing by reading a Jira ticket, syncing local repos, finding pmm-submodules build artifacts, analyzing flaky FB Tests for test insights, choosing pmm3-aws-staging-start or pmm3-deploy-services, determining all Jenkins parameters, generating a fully pre-filled parambuild URL, screenshotting FB test results with playwright-cli into Jira FB test screenshots field, and writing verified test steps. Use when the user asks for help with manual testing, PMM staging setup, FB test analysis, or wants a Jenkins job link to test a Jira ticket.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Source

percona

percona/pmm-qa

View GitHub Repository View Creator Repositories

Download

Related occupationsSOC

Based on SOC occupation classification

Software Quality Assurance Analysts and TestersComputer and Mathematical Occupations·SOC 15-1253

File Explorer

3 files

SKILL.md

readonly

name

pmm-manual-test

description

PMM Manual Test

Trigger

Start when the user asks for help with manual testing. Always ask for the Jira ticket key (e.g. PMM-14915) if not provided.

Local repo layout

PMM repos are expected as siblings under one parent directory (repos root). This skill ships in pmm-qa; resolve paths from the pmm-qa clone.

Repo	Directory (under repos root)	Remote
pmm-qa	`pmm-qa`	`percona/pmm-qa`
pmm	`pmm`	`percona/pmm`
grafana	`grafana`	`percona/grafana`
jenkins-pipelines	`jenkins-pipelines`	`Percona-Lab/jenkins-pipelines`

pmm-submodules — read PR comments and checks via gh only, never clone.

Resolve repos root and paths

Repos root = parent directory of the pmm-qa git root.

$pmmQaRoot = git -C "<path-to-pmm-qa>" rev-parse --show-toplevel
$ReposRoot = (Get-Item $pmmQaRoot).Parent.FullName

PMM_QA_ROOT="$(git -C pmm-qa rev-parse --show-toplevel)"
REPOS_ROOT="$(dirname "$PMM_QA_ROOT")"

Variable	Example path
`$ReposRoot` / `$REPOS_ROOT`	`~/vscodeProjects/PMM`
pmm	`$ReposRoot/pmm`
grafana	`$ReposRoot/grafana`
jenkins-pipelines	`$ReposRoot/jenkins-pipelines`
pmm-qa	`$ReposRoot/pmm-qa`

Clone if missing

If a sibling repo directory does not exist, clone it into repos root:

$ReposRoot = (Get-Item (git rev-parse --show-toplevel)).Parent.FullName
$repos = @{
  pmm = "https://github.com/percona/pmm.git"
  grafana = "https://github.com/percona/grafana.git"
  "jenkins-pipelines" = "https://github.com/Percona-Lab/jenkins-pipelines.git"
}
foreach ($name in $repos.Keys) {
  $path = Join-Path $ReposRoot $name
  if (-not (Test-Path $path)) { git clone $repos[$name] $path }
}

REPOS_ROOT="$(dirname "$(git rev-parse --show-toplevel)")"
[ -d "$REPOS_ROOT/pmm" ] || git clone https://github.com/percona/pmm.git "$REPOS_ROOT/pmm"
[ -d "$REPOS_ROOT/grafana" ] || git clone https://github.com/percona/grafana.git "$REPOS_ROOT/grafana"
[ -d "$REPOS_ROOT/jenkins-pipelines" ] || git clone https://github.com/Percona-Lab/jenkins-pipelines.git "$REPOS_ROOT/jenkins-pipelines"

Then git fetch / git pull --ff-only in each repo that exists.

Workflow checklist

- [ ] 1. Get Jira ticket key from user
- [ ] 2. Read ticket via Atlassian MCP (summary, description, linked PRs, acceptance criteria)
- [ ] 3. Identify affected repos (pmm, grafana, or both)
- [ ] 4. Resolve repos root; clone missing siblings; update local repos (git pull)
- [ ] 5. Find pmm-submodules PR and latest JNKPercona build comment — older comments are not valid
- [ ] 6. Analyze FB Tests from latest FB build (`gh pr checks` on pmm-submodules PR)
- [ ] 7. Choose Jenkins job (pmm3-aws-staging-start vs pmm3-deploy-services)
- [ ] 8. Determine required job parameters to setup instance based on ticket scope and PR diffs
- [ ] 9. Build complete parambuild URL (all relevant params, not just DOCKER_VERSION + CLIENT_VERSION)
- [ ] 10. Review PR diffs — do not trust "How to test" blindly
- [ ] 11. Write test instructions in chat (adapt using FB test failures where relevant)
- [ ] 12. Screenshot FB Tests Actions run (playwright-cli) and update Jira FB test screenshots field
- [ ] 13. Open Jenkins parambuild URL in browser

Step 1: Get Jira ticket key

Ask if missing. Proceed once you have the key (e.g. PMM-14915).

Step 2: Read Jira ticket

Use Atlassian MCP (jira_get_issue, jira_get_issue_development_info):

Summary, description, QA notes, acceptance criteria
Existing How to test (customfield_10083) and FB test screenshots (customfield_10492)
Development panel: linked GitHub PRs/branches
Environment or setup hints

Setup is ticket-specific. Pay attention to:

Which subsystems change (server, UI/grafana, clients, telemetry, advisors, etc.)
Whether monitored databases are needed
Required PMM server environment variables (DOCKER_ENV_VARIABLE) — job defaults are often wrong for the test (e.g. default PMM_ENABLE_TELEMETRY=0)
PMM Settings changes the user must apply after provisioning

Step 3: Identify affected repos

Change area	Repo
Backend, API, managed services	`percona/pmm`
Grafana UI, dashboards, frontend	`percona/grafana`
Both	Check both PRs

Grafana-only tickets still need an FB server image from pmm-submodules.

Step 4: Update local repos

Resolve $ReposRoot (see above). Clone any missing sibling, then pull:

git -C "$ReposRoot/pmm" fetch origin
git -C "$ReposRoot/pmm" pull --ff-only

Repeat for grafana, jenkins-pipelines, and pmm-qa as needed.

Step 5: Find linked PRs and build artifacts

Find PRs

gh search prs "PMM-14915" --repo percona/pmm --json number,title,url
gh search prs "PMM-14915" --repo percona/grafana --json number,title,url

gh pr view <n> --repo percona/pmm --json title,body,url,files
gh pr diff <n> --repo percona/pmm

pmm-submodules PR

Read the FB link in the pmm PR body (e.g. Percona-Lab/pmm-submodules/pull/4376). Submodules PR number ≠ pmm PR number.

JNKPercona build comment

gh api repos/Percona-Lab/pmm-submodules/issues/<SUBMODULES_PR>/comments `
  --jq '[.[] | select(.user.login == "JNKPercona" and (.body | contains("Staging instance:"))) | {created_at, body}] | sort_by(.created_at) | .[-1]'

Extract from the latest build-summary comment (older FB builds are invalid):

Field	Param
`Server docker:`	`DOCKER_VERSION`
`Watchtower docker:`	`WATCHTOWER_VERSION`
`Client tarball:`	`CLIENT_VERSION`
`Client docker:`	ignore for `CLIENT_VERSION`

Step 6: Analyze FB Tests

FB Tests = GitHub PR checks on pmm-submodules (gh pr checks). They are often flaky but give useful hints — use failures to adapt manual test steps when they overlap the ticket scope.

Ignore JNKPercona API tests have succeded/failed comments — not relevant for this workflow.

Collect results

gh pr checks <SUBMODULES_PR> -R Percona-Lab/pmm-submodules

What to extract

Source	Data
`gh pr checks`	Failed/passed UI (`@* UI tests`) and CLI (`CLI tests *`) matrix
Check links	Actions run URL, Jenkins `pmm3-submodules/PR-XXXX` build

How to use failures

List failures from the latest FB build only
Mark each: relevant (overlaps ticket) / flaky / out of scope
Relevant failures → add explicit checks to manual test plan (step 11) and Jenkins params (step 8) if setup-related
Flaky/out of scope → note in FB test screenshots field, do not expand manual scope

Full reference: fb-tests.md

Step 7: Choose Jenkins job

Scenario	Job
Server/UI/API; optional local clients on same VM via `CLIENTS`	pmm3-aws-staging-start
Separate client VMs, multi-node DB topologies, external exporters	pmm3-deploy-services

FB test failures may inform this choice (e.g. CLI integration failure on a DB type → consider CLIENTS or DEPLOY_*_GROUP).

staging-start with local clients (`CLIENTS`)

CLIENTS feeds pmm-framework.py in this repo (qa-integration/pmm_qa/pmm-framework.py) on the same EC2 instance.

Limitations: single VM only; no multi-VM topologies. Use deploy-services for those.

Leave CLIENTS empty for server-only tests (UI, API, settings, network-tab checks).

See jenkins-jobs.md for all parameters. Groovy source: $ReposRoot/jenkins-pipelines/pmm/v3/.

Step 8: Determine required job parameters

Derive every param that differs from defaults. The user should not need to edit the Jenkins form.

Include all params in the parambuild URL for the chosen job — not only DOCKER_VERSION + CLIENT_VERSION. Walk through the full list below; keep defaults only when they are correct for this ticket.

Always set from FB build

DOCKER_VERSION, CLIENT_VERSION, WATCHTOWER_VERSION

Adapt from ticket + PR diffs + FB test insights

Concern	Params to review
PMM server env vars	`DOCKER_ENV_VARIABLE` — read PR diff + `$ReposRoot/pmm/managed/CONTRIBUTING.md` + docker env docs
Monitored databases	`CLIENTS` (staging-start) or `DEPLOY_*_GROUP` flags (deploy-services)
DB versions	`PS_VERSION`, `MS_VERSION`, `PXC_VERSION`, `PGSQL_VERSION`, `PDPGSQL_VERSION`, `PSMDB_VERSION`, `MODB_VERSION`
SSH access	`SSH_KEY`
Client repo channel	`PMM_CLIENT_REPO` (staging-start only)
Provisioner scripts	`PMM_QA_GIT_BRANCH`
Instance lifetime	`DAYS` (staging-start)
Pull mode	`ENABLE_PULL_MODE`
Server install type	`SERVER_TYPE` (deploy-services only)
FB failure pointed at specific suite	Match setup (e.g. PSMDB failure → `--database psmdb` or `DEPLOY_MONGO_GROUP=true`)

pmm3-aws-staging-start — review every param

Param	Default	When to override
`DOCKER_VERSION`	`perconalab/pmm-server:3-dev-latest`	Always → FB server image
`WATCHTOWER_VERSION`	`perconalab/watchtower:dev-latest`	Always → FB watchtower image
`CLIENT_VERSION`	`3-dev-latest`	Always → FB client tarball URL
`SSH_KEY`	(empty)	When SSH access to EC2 is needed
`ADMIN_PASSWORD`	`pmm3admin!`	When test docs use a different password
`PMM_CLIENT_REPO`	`experimental`	`testing` / `release` for RC or release client tests
`ENABLE_PULL_MODE`	`no`	`yes` when testing pull-mode clients
`DAYS`	`1`	`0` to disable auto-stop; higher if long test
`DOCKER_ENV_VARIABLE`	`-e PMM_DEBUG=1 -e PMM_ENABLE_TELEMETRY=0`	Often wrong by default — set per ticket
`PXC_VERSION`	`8.4`	When `CLIENTS` uses `pxc`
`PS_VERSION`	`8.0`	When `CLIENTS` uses `ps`
`MS_VERSION`	`8.0`	When `CLIENTS` uses `mysql`
`PGSQL_VERSION`	`16`	When `CLIENTS` uses `pgsql`
`PDPGSQL_VERSION`	`16`	When `CLIENTS` uses `pdpgsql`
`PSMDB_VERSION`	`8.0`	When `CLIENTS` uses `psmdb`
`MODB_VERSION`	`8.0`	When `CLIENTS` uses `modb`
`CLIENTS`	`--database ps=5.7,...`	Empty for server-only; set `--database ...` for local clients
`NOTIFY`	`false`	Usually keep `false`
`CLIENT_INSTANCE`	`no`	`yes` only when this VM is a client node for another server
`SERVER_IP`	`0.0.0.0`	Required when `CLIENT_INSTANCE=yes`
`PMM_QA_GIT_BRANCH`	`main`	Branch with provisioner fixes for the test

pmm3-deploy-services — review every param

Param	Default	When to override
`SERVER_TYPE`	`docker`	`ovf` / `ami` / `helm` / `ha` when ticket tests those install paths
`DOCKER_VERSION`	`perconalab/pmm-server:3-dev-latest`	Always → FB server image (docker/helm/ha)
`OVA_VERSION`	`PMM3-Server-OVF-3.0.0-latest.ova`	When `SERVER_TYPE=ovf`
`AMI_ID`	`ami-0669b163befffb6c3`	When `SERVER_TYPE=ami`
`CLIENT_VERSION`	`3-dev-latest`	Always → FB client tarball URL
`ENABLE_PULL_MODE`	`no`	`yes` for pull-mode client tests
`ADMIN_PASSWORD`	`pmm3admin!`	When test requires specific password
`SSH_KEY`	(empty)	When SSH access to client/server VMs is needed
`DEPLOY_EXTERNAL`	`false`	`true` for external node + HAProxy tests
`DEPLOY_MYSQL_GROUP`	`false`	`true` for GR, PXC, async repl, standalone MySQL VMs
`DEPLOY_MONGO_GROUP`	`false`	`true` for sharded + PSS Mongo VMs
`DEPLOY_POSTGRES_GROUP`	`false`	`true` for PG, PDPGSQL, Patroni VMs
`DEPLOY_VALKEY`	`false`	`true` for Valkey tests
`GENERATE_DASHBOARD_SCREENSHOTS`	`false`	`true` only when screenshot generation is the goal
`SCREENSHOTS_SLACK_TARGET`	`@catalina.adam`	When screenshots enabled
`PXC_VERSION`	`8.0`	Match DB versions under test
`PS_VERSION`	`8.4`
`MS_VERSION`	`8.4`
`PGSQL_VERSION`	`17`
`PDPGSQL_VERSION`	`17`
`PSMDB_VERSION`	`8.0`
`MODB_VERSION`	`8.0`
`HELM_CHART_BRANCH`	`pmmha-v3`	When `SERVER_TYPE=ha`
`OPENSHIFT_VERSION`	`latest`	When `SERVER_TYPE=helm`
`K8S_VERSION`	`1.34`	When `SERVER_TYPE=ha`
`WATCHTOWER_VERSION`	`perconalab/watchtower:dev-latest`	Always → FB watchtower image
`PMM_QA_GIT_BRANCH`	`main`	Branch with provisioner fixes

Note: deploy-services nested runStagingServer uses a fixed DOCKER_ENV_VARIABLE. For env-var-heavy tickets, prefer staging-start unless multi-VM clients are required.

`DOCKER_ENV_VARIABLE` — common mistake

Job default (staging-start): -e PMM_DEBUG=1 -e PMM_ENABLE_TELEMETRY=0 — override when the ticket requires different server config.

Full defaults and CLIENTS syntax: jenkins-jobs.md.

Step 9: Build complete parambuild URL

Pre-fill all parameters from step 8 — every param from the chosen job table that applies.

Base URLs:

staging-start: https://pmm.cd.percona.com/job/pmm3-aws-staging-start/parambuild/
deploy-services: https://pmm.cd.percona.com/job/pmm3-deploy-services/parambuild/

URL-encode DOCKER_ENV_VARIABLE and CLIENTS. Full reference: jenkins-jobs.md.

Step 10: Review PR diffs

Do not trust PR "How to test" or Jira QA bullets without verifying code.

Read diffs in percona/pmm and/or percona/grafana
Map changes to concrete UI/API/network steps
Cross-check with FB test failures marked relevant
Note post-provision steps (PMM Settings toggles, etc.)

Step 11: Write test instructions

Post in chat before Jira update and browser:

Job chosen and why
Params set and why
FB test summary (failures relevant to this ticket)
Provision → configure → exercise → expected result
DevTools / logs / PMM Settings checks

Step 12: Screenshot FB Tests run → Jira

When to screenshot

Attach a screenshot to Jira only when all gh pr checks are green (no fail lines).

If any check failed: skip screenshot; text-only update to customfield_10492.

If all passed: screenshot the FB Tests Actions run:

$runId = gh pr checks <SUBMODULES_PR> -R Percona-Lab/pmm-submodules 2>&1 |
  Select-String -Pattern 'actions/runs/(\d+)' |
  ForEach-Object { $_.Matches[0].Groups[1].Value } |
  Select-Object -First 1

playwright-cli open "https://github.com/Percona-Lab/pmm-submodules/actions/runs/$runId"
playwright-cli resize 1920 1080
playwright-cli screenshot --filename="fb-test-<TICKET>-checks.png"
playwright-cli close

Jira fields: customfield_10492 (FB test screenshots), customfield_10083 (How to test).

Ask the user before writing to Jira unless they explicitly requested the update.

Details: fb-tests.md

Step 13: Open browser

Start-Process "<full-parambuild-url>"

gh CLI quick reference

Task	Command
FB Tests (PR checks)	`gh pr checks <n> -R Percona-Lab/pmm-submodules`
View PR	`gh pr view <n> -R owner/repo`
PR diff	`gh pr diff <n> -R owner/repo`
Search by ticket	`gh search prs "PMM-XXXX" --repo percona/pmm`
PR comments	`gh api repos/OWNER/REPO/issues/<n>/comments`

Done criteria

Jira read; scope understood
Latest JNKPercona build comment used
FB Tests analyzed; relevant failures reflected in manual plan
Correct job + complete parambuild URL
Test instructions posted in chat
Jira FB test screenshots updated — screenshot only if all checks passed (when user approves)
Browser opened to parambuild URL

Additional resources

jenkins-jobs.md — all Jenkins parameters
fb-tests.md — FB test sources, flaky guidance, screenshot & Jira templates

name

pmm-manual-test

description

PMM Manual Test

Trigger

Start when the user asks for help with manual testing. Always ask for the Jira ticket key (e.g. PMM-14915) if not provided.

Local repo layout

PMM repos are expected as siblings under one parent directory (repos root). This skill ships in pmm-qa; resolve paths from the pmm-qa clone.

Repo	Directory (under repos root)	Remote
pmm-qa	`pmm-qa`	`percona/pmm-qa`
pmm	`pmm`	`percona/pmm`
grafana	`grafana`	`percona/grafana`
jenkins-pipelines	`jenkins-pipelines`	`Percona-Lab/jenkins-pipelines`

pmm-submodules — read PR comments and checks via gh only, never clone.

Resolve repos root and paths

Repos root = parent directory of the pmm-qa git root.

$pmmQaRoot = git -C "<path-to-pmm-qa>" rev-parse --show-toplevel
$ReposRoot = (Get-Item $pmmQaRoot).Parent.FullName

PMM_QA_ROOT="$(git -C pmm-qa rev-parse --show-toplevel)"
REPOS_ROOT="$(dirname "$PMM_QA_ROOT")"

Variable	Example path
`$ReposRoot` / `$REPOS_ROOT`	`~/vscodeProjects/PMM`
pmm	`$ReposRoot/pmm`
grafana	`$ReposRoot/grafana`
jenkins-pipelines	`$ReposRoot/jenkins-pipelines`
pmm-qa	`$ReposRoot/pmm-qa`

Clone if missing

If a sibling repo directory does not exist, clone it into repos root:

$ReposRoot = (Get-Item (git rev-parse --show-toplevel)).Parent.FullName
$repos = @{
  pmm = "https://github.com/percona/pmm.git"
  grafana = "https://github.com/percona/grafana.git"
  "jenkins-pipelines" = "https://github.com/Percona-Lab/jenkins-pipelines.git"
}
foreach ($name in $repos.Keys) {
  $path = Join-Path $ReposRoot $name
  if (-not (Test-Path $path)) { git clone $repos[$name] $path }
}

REPOS_ROOT="$(dirname "$(git rev-parse --show-toplevel)")"
[ -d "$REPOS_ROOT/pmm" ] || git clone https://github.com/percona/pmm.git "$REPOS_ROOT/pmm"
[ -d "$REPOS_ROOT/grafana" ] || git clone https://github.com/percona/grafana.git "$REPOS_ROOT/grafana"
[ -d "$REPOS_ROOT/jenkins-pipelines" ] || git clone https://github.com/Percona-Lab/jenkins-pipelines.git "$REPOS_ROOT/jenkins-pipelines"

Then git fetch / git pull --ff-only in each repo that exists.

Workflow checklist

- [ ] 1. Get Jira ticket key from user
- [ ] 2. Read ticket via Atlassian MCP (summary, description, linked PRs, acceptance criteria)
- [ ] 3. Identify affected repos (pmm, grafana, or both)
- [ ] 4. Resolve repos root; clone missing siblings; update local repos (git pull)
- [ ] 5. Find pmm-submodules PR and latest JNKPercona build comment — older comments are not valid
- [ ] 6. Analyze FB Tests from latest FB build (`gh pr checks` on pmm-submodules PR)
- [ ] 7. Choose Jenkins job (pmm3-aws-staging-start vs pmm3-deploy-services)
- [ ] 8. Determine required job parameters to setup instance based on ticket scope and PR diffs
- [ ] 9. Build complete parambuild URL (all relevant params, not just DOCKER_VERSION + CLIENT_VERSION)
- [ ] 10. Review PR diffs — do not trust "How to test" blindly
- [ ] 11. Write test instructions in chat (adapt using FB test failures where relevant)
- [ ] 12. Screenshot FB Tests Actions run (playwright-cli) and update Jira FB test screenshots field
- [ ] 13. Open Jenkins parambuild URL in browser

Step 1: Get Jira ticket key

Ask if missing. Proceed once you have the key (e.g. PMM-14915).

Step 2: Read Jira ticket

Use Atlassian MCP (jira_get_issue, jira_get_issue_development_info):

Summary, description, QA notes, acceptance criteria
Existing How to test (customfield_10083) and FB test screenshots (customfield_10492)
Development panel: linked GitHub PRs/branches
Environment or setup hints

Setup is ticket-specific. Pay attention to:

Which subsystems change (server, UI/grafana, clients, telemetry, advisors, etc.)
Whether monitored databases are needed
Required PMM server environment variables (DOCKER_ENV_VARIABLE) — job defaults are often wrong for the test (e.g. default PMM_ENABLE_TELEMETRY=0)
PMM Settings changes the user must apply after provisioning

Step 3: Identify affected repos

Change area	Repo
Backend, API, managed services	`percona/pmm`
Grafana UI, dashboards, frontend	`percona/grafana`
Both	Check both PRs

Grafana-only tickets still need an FB server image from pmm-submodules.

Step 4: Update local repos

Resolve $ReposRoot (see above). Clone any missing sibling, then pull:

git -C "$ReposRoot/pmm" fetch origin
git -C "$ReposRoot/pmm" pull --ff-only

Repeat for grafana, jenkins-pipelines, and pmm-qa as needed.

Step 5: Find linked PRs and build artifacts

Find PRs

gh search prs "PMM-14915" --repo percona/pmm --json number,title,url
gh search prs "PMM-14915" --repo percona/grafana --json number,title,url

gh pr view <n> --repo percona/pmm --json title,body,url,files
gh pr diff <n> --repo percona/pmm

pmm-submodules PR

Read the FB link in the pmm PR body (e.g. Percona-Lab/pmm-submodules/pull/4376). Submodules PR number ≠ pmm PR number.

JNKPercona build comment

gh api repos/Percona-Lab/pmm-submodules/issues/<SUBMODULES_PR>/comments `
  --jq '[.[] | select(.user.login == "JNKPercona" and (.body | contains("Staging instance:"))) | {created_at, body}] | sort_by(.created_at) | .[-1]'

Extract from the latest build-summary comment (older FB builds are invalid):

Field	Param
`Server docker:`	`DOCKER_VERSION`
`Watchtower docker:`	`WATCHTOWER_VERSION`
`Client tarball:`	`CLIENT_VERSION`
`Client docker:`	ignore for `CLIENT_VERSION`

Step 6: Analyze FB Tests

Ignore JNKPercona API tests have succeded/failed comments — not relevant for this workflow.

Collect results

gh pr checks <SUBMODULES_PR> -R Percona-Lab/pmm-submodules

What to extract

Source	Data
`gh pr checks`	Failed/passed UI (`@* UI tests`) and CLI (`CLI tests *`) matrix
Check links	Actions run URL, Jenkins `pmm3-submodules/PR-XXXX` build

How to use failures

List failures from the latest FB build only
Mark each: relevant (overlaps ticket) / flaky / out of scope
Relevant failures → add explicit checks to manual test plan (step 11) and Jenkins params (step 8) if setup-related
Flaky/out of scope → note in FB test screenshots field, do not expand manual scope

Full reference: fb-tests.md

Step 7: Choose Jenkins job

Scenario	Job
Server/UI/API; optional local clients on same VM via `CLIENTS`	pmm3-aws-staging-start
Separate client VMs, multi-node DB topologies, external exporters	pmm3-deploy-services

FB test failures may inform this choice (e.g. CLI integration failure on a DB type → consider CLIENTS or DEPLOY_*_GROUP).

staging-start with local clients (`CLIENTS`)

CLIENTS feeds pmm-framework.py in this repo (qa-integration/pmm_qa/pmm-framework.py) on the same EC2 instance.

Limitations: single VM only; no multi-VM topologies. Use deploy-services for those.

Leave CLIENTS empty for server-only tests (UI, API, settings, network-tab checks).

See jenkins-jobs.md for all parameters. Groovy source: $ReposRoot/jenkins-pipelines/pmm/v3/.

Step 8: Determine required job parameters

Derive every param that differs from defaults. The user should not need to edit the Jenkins form.

Always set from FB build

DOCKER_VERSION, CLIENT_VERSION, WATCHTOWER_VERSION

Adapt from ticket + PR diffs + FB test insights

Concern	Params to review
PMM server env vars	`DOCKER_ENV_VARIABLE` — read PR diff + `$ReposRoot/pmm/managed/CONTRIBUTING.md` + docker env docs
Monitored databases	`CLIENTS` (staging-start) or `DEPLOY_*_GROUP` flags (deploy-services)
DB versions	`PS_VERSION`, `MS_VERSION`, `PXC_VERSION`, `PGSQL_VERSION`, `PDPGSQL_VERSION`, `PSMDB_VERSION`, `MODB_VERSION`
SSH access	`SSH_KEY`
Client repo channel	`PMM_CLIENT_REPO` (staging-start only)
Provisioner scripts	`PMM_QA_GIT_BRANCH`
Instance lifetime	`DAYS` (staging-start)
Pull mode	`ENABLE_PULL_MODE`
Server install type	`SERVER_TYPE` (deploy-services only)
FB failure pointed at specific suite	Match setup (e.g. PSMDB failure → `--database psmdb` or `DEPLOY_MONGO_GROUP=true`)

pmm3-aws-staging-start — review every param

Param	Default	When to override
`DOCKER_VERSION`	`perconalab/pmm-server:3-dev-latest`	Always → FB server image
`WATCHTOWER_VERSION`	`perconalab/watchtower:dev-latest`	Always → FB watchtower image
`CLIENT_VERSION`	`3-dev-latest`	Always → FB client tarball URL
`SSH_KEY`	(empty)	When SSH access to EC2 is needed
`ADMIN_PASSWORD`	`pmm3admin!`	When test docs use a different password
`PMM_CLIENT_REPO`	`experimental`	`testing` / `release` for RC or release client tests
`ENABLE_PULL_MODE`	`no`	`yes` when testing pull-mode clients
`DAYS`	`1`	`0` to disable auto-stop; higher if long test
`DOCKER_ENV_VARIABLE`	`-e PMM_DEBUG=1 -e PMM_ENABLE_TELEMETRY=0`	Often wrong by default — set per ticket
`PXC_VERSION`	`8.4`	When `CLIENTS` uses `pxc`
`PS_VERSION`	`8.0`	When `CLIENTS` uses `ps`
`MS_VERSION`	`8.0`	When `CLIENTS` uses `mysql`
`PGSQL_VERSION`	`16`	When `CLIENTS` uses `pgsql`
`PDPGSQL_VERSION`	`16`	When `CLIENTS` uses `pdpgsql`
`PSMDB_VERSION`	`8.0`	When `CLIENTS` uses `psmdb`
`MODB_VERSION`	`8.0`	When `CLIENTS` uses `modb`
`CLIENTS`	`--database ps=5.7,...`	Empty for server-only; set `--database ...` for local clients
`NOTIFY`	`false`	Usually keep `false`
`CLIENT_INSTANCE`	`no`	`yes` only when this VM is a client node for another server
`SERVER_IP`	`0.0.0.0`	Required when `CLIENT_INSTANCE=yes`
`PMM_QA_GIT_BRANCH`	`main`	Branch with provisioner fixes for the test

pmm3-deploy-services — review every param

Param	Default	When to override
`SERVER_TYPE`	`docker`	`ovf` / `ami` / `helm` / `ha` when ticket tests those install paths
`DOCKER_VERSION`	`perconalab/pmm-server:3-dev-latest`	Always → FB server image (docker/helm/ha)
`OVA_VERSION`	`PMM3-Server-OVF-3.0.0-latest.ova`	When `SERVER_TYPE=ovf`
`AMI_ID`	`ami-0669b163befffb6c3`	When `SERVER_TYPE=ami`
`CLIENT_VERSION`	`3-dev-latest`	Always → FB client tarball URL
`ENABLE_PULL_MODE`	`no`	`yes` for pull-mode client tests
`ADMIN_PASSWORD`	`pmm3admin!`	When test requires specific password
`SSH_KEY`	(empty)	When SSH access to client/server VMs is needed
`DEPLOY_EXTERNAL`	`false`	`true` for external node + HAProxy tests
`DEPLOY_MYSQL_GROUP`	`false`	`true` for GR, PXC, async repl, standalone MySQL VMs
`DEPLOY_MONGO_GROUP`	`false`	`true` for sharded + PSS Mongo VMs
`DEPLOY_POSTGRES_GROUP`	`false`	`true` for PG, PDPGSQL, Patroni VMs
`DEPLOY_VALKEY`	`false`	`true` for Valkey tests
`GENERATE_DASHBOARD_SCREENSHOTS`	`false`	`true` only when screenshot generation is the goal
`SCREENSHOTS_SLACK_TARGET`	`@catalina.adam`	When screenshots enabled
`PXC_VERSION`	`8.0`	Match DB versions under test
`PS_VERSION`	`8.4`
`MS_VERSION`	`8.4`
`PGSQL_VERSION`	`17`
`PDPGSQL_VERSION`	`17`
`PSMDB_VERSION`	`8.0`
`MODB_VERSION`	`8.0`
`HELM_CHART_BRANCH`	`pmmha-v3`	When `SERVER_TYPE=ha`
`OPENSHIFT_VERSION`	`latest`	When `SERVER_TYPE=helm`
`K8S_VERSION`	`1.34`	When `SERVER_TYPE=ha`
`WATCHTOWER_VERSION`	`perconalab/watchtower:dev-latest`	Always → FB watchtower image
`PMM_QA_GIT_BRANCH`	`main`	Branch with provisioner fixes

Note: deploy-services nested runStagingServer uses a fixed DOCKER_ENV_VARIABLE. For env-var-heavy tickets, prefer staging-start unless multi-VM clients are required.

`DOCKER_ENV_VARIABLE` — common mistake

Job default (staging-start): -e PMM_DEBUG=1 -e PMM_ENABLE_TELEMETRY=0 — override when the ticket requires different server config.

Full defaults and CLIENTS syntax: jenkins-jobs.md.

Step 9: Build complete parambuild URL

Pre-fill all parameters from step 8 — every param from the chosen job table that applies.

Base URLs:

staging-start: https://pmm.cd.percona.com/job/pmm3-aws-staging-start/parambuild/
deploy-services: https://pmm.cd.percona.com/job/pmm3-deploy-services/parambuild/

URL-encode DOCKER_ENV_VARIABLE and CLIENTS. Full reference: jenkins-jobs.md.

Step 10: Review PR diffs

Do not trust PR "How to test" or Jira QA bullets without verifying code.

Read diffs in percona/pmm and/or percona/grafana
Map changes to concrete UI/API/network steps
Cross-check with FB test failures marked relevant
Note post-provision steps (PMM Settings toggles, etc.)

Step 11: Write test instructions

Post in chat before Jira update and browser:

Job chosen and why
Params set and why
FB test summary (failures relevant to this ticket)
Provision → configure → exercise → expected result
DevTools / logs / PMM Settings checks

Step 12: Screenshot FB Tests run → Jira

When to screenshot

Attach a screenshot to Jira only when all gh pr checks are green (no fail lines).

If any check failed: skip screenshot; text-only update to customfield_10492.

If all passed: screenshot the FB Tests Actions run:

$runId = gh pr checks <SUBMODULES_PR> -R Percona-Lab/pmm-submodules 2>&1 |
  Select-String -Pattern 'actions/runs/(\d+)' |
  ForEach-Object { $_.Matches[0].Groups[1].Value } |
  Select-Object -First 1

playwright-cli open "https://github.com/Percona-Lab/pmm-submodules/actions/runs/$runId"
playwright-cli resize 1920 1080
playwright-cli screenshot --filename="fb-test-<TICKET>-checks.png"
playwright-cli close

Jira fields: customfield_10492 (FB test screenshots), customfield_10083 (How to test).

Ask the user before writing to Jira unless they explicitly requested the update.

Details: fb-tests.md

Step 13: Open browser

Start-Process "<full-parambuild-url>"

gh CLI quick reference

Task	Command
FB Tests (PR checks)	`gh pr checks <n> -R Percona-Lab/pmm-submodules`
View PR	`gh pr view <n> -R owner/repo`
PR diff	`gh pr diff <n> -R owner/repo`
Search by ticket	`gh search prs "PMM-XXXX" --repo percona/pmm`
PR comments	`gh api repos/OWNER/REPO/issues/<n>/comments`

Done criteria

Jira read; scope understood
Latest JNKPercona build comment used
FB Tests analyzed; relevant failures reflected in manual plan
Correct job + complete parambuild URL
Test instructions posted in chat
Jira FB test screenshots updated — screenshot only if all checks passed (when user approves)
Browser opened to parambuild URL

Additional resources

jenkins-jobs.md — all Jenkins parameters
fb-tests.md — FB test sources, flaky guidance, screenshot & Jira templates

pmm-manual-test

PMM Manual Test

Trigger

Local repo layout

Resolve repos root and paths

Clone if missing

Workflow checklist

Step 1: Get Jira ticket key

Step 2: Read Jira ticket

Step 3: Identify affected repos

Step 4: Update local repos

Step 5: Find linked PRs and build artifacts

Find PRs

pmm-submodules PR

JNKPercona build comment

Step 6: Analyze FB Tests

Collect results

What to extract

How to use failures

Step 7: Choose Jenkins job

staging-start with local clients (CLIENTS)

Step 8: Determine required job parameters

Always set from FB build

Adapt from ticket + PR diffs + FB test insights

pmm3-aws-staging-start — review every param

pmm3-deploy-services — review every param

DOCKER_ENV_VARIABLE — common mistake

Step 9: Build complete parambuild URL

Step 10: Review PR diffs

Step 11: Write test instructions

Step 12: Screenshot FB Tests run → Jira

When to screenshot

Step 13: Open browser

gh CLI quick reference

Done criteria

Additional resources

PMM Manual Test

Trigger

Local repo layout

Resolve repos root and paths

Clone if missing

Workflow checklist

Step 1: Get Jira ticket key

Step 2: Read Jira ticket

Step 3: Identify affected repos

Step 4: Update local repos

Step 5: Find linked PRs and build artifacts

Find PRs

pmm-submodules PR

JNKPercona build comment

Step 6: Analyze FB Tests

Collect results

What to extract

How to use failures

Step 7: Choose Jenkins job

staging-start with local clients (CLIENTS)

Step 8: Determine required job parameters

Always set from FB build

Adapt from ticket + PR diffs + FB test insights

pmm3-aws-staging-start — review every param

pmm3-deploy-services — review every param

DOCKER_ENV_VARIABLE — common mistake

Step 9: Build complete parambuild URL

Step 10: Review PR diffs

Step 11: Write test instructions

Step 12: Screenshot FB Tests run → Jira

When to screenshot

Step 13: Open browser

gh CLI quick reference

Done criteria

Additional resources

staging-start with local clients (`CLIENTS`)

`DOCKER_ENV_VARIABLE` — common mistake

staging-start with local clients (`CLIENTS`)

`DOCKER_ENV_VARIABLE` — common mistake