القائمة
How to write behave scenarios and step definitions for the testsuite repo. Load when editing .feature files or steps.py.
يُحدَّد Prompt الذي يراجع المصدر أولًا بشكل افتراضي. يمكنك التبديل إلى أمر مباشر أو تنزيل نسخة محلية.
اقرأ SKILL.md وأي ملفات مرافقة يعرضها SkillsMP قبل أن تقرر التثبيت.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
يتجاوز الأمر المباشر Prompt المخصّص للمراجعة. افحص المصدر قبل تشغيله.
npx skills add https://github.com/projectbluefin/testsuite --skill behaveيبقى الأمر في سطر واحد. مرّر أفقيًا لمراجعته كاملًا قبل النسخ.
تفضّل نسخة محلية؟ نزّل الملفات المتاحة حاليًا لدى SkillsMP.
Task → skill router and hard-rule set for projectbluefin/testsuite. Load this first in every session, then load only the sub-skills your task needs.
How to call and debug the reusable testsuite e2e workflow. Load when changing e2e.yml, action inputs, or consumer-repo wiring.
Authoritative coverage matrix and @future gap list. Load during planning, coverage reviews, or when adding a new suite.
عرض SKILL.md
| name | behave |
| version | 1.0 |
| last_updated | 2026-07-29 |
| id | behave |
| one_line_purpose | Write behave scenarios and step definitions for testsuite suites. |
| entry_point | docs/skills/test-authoring/behave/SKILL.md |
| category | test-authoring |
| mcp_compliance_level | partial |
| status | active |
| dependencies | [] |
| tags | ["behave","bdd","steps"] |
| description | How to write behave scenarios and step definitions for the testsuite repo. Load when editing .feature files or steps.py. |
| metadata | {"type":"pattern","audience":"agents","maturity":"stable"} |
.feature files in any suiteUndefinedStep, AmbiguousStep, or dry-run failuresdocs/skills/test-authoring/gnome/SKILL.mddocs/skills/test-authoring/bootc/SKILL.md.feature file and the suite's steps/*.py before adding phrases.tests/shared/ssh_steps.py for generic SSH command/assertion steps instead of duplicating helpers.tests/shared/ssh_steps obligates the suite to set the context attributes those steps read. run_ssh() dereferences context.ssh_key, context.ssh_user, context.vm_ip and (optionally) context.ssh_port; a suite that skips this fails every SSH scenario with AttributeError on the first step. Call tests.shared.ssh_config.populate_ssh_context(context) from the suite's before_all (see tests/software/features/environment.py) — it resolves context attributes → behave userdata → SSH_KEY/VM_IP/VM_USER/SSH_PORT env vars → runner defaults, which is the same source suite-local SSH helpers (e.g. the software suite's _flatpak()) must use. Never let a suite keep a second, env-only SSH path alongside the shared steps.behave --dry-run on the touched suite before pushing so undefined or ambiguous phrases fail locally.grep -q vs grep -c for existence checksNever use for existence checks in SSH commands. This is a false-positive trap:
grep -c ... || echo 0grep -c prints 0 and exits 1, then echo 0 fires and the overall command exits 0"0\n0" — the step SSH command output is not "0" sees "0\n0" which is not equal to "0" and passes falselyCorrect pattern — use grep -q and assert the return code:
Scenario: Portal interface is present
* Run SSH command: "gdbus introspect ... | grep -q 'InterfaceName'"
* SSH command return code is "0"
grep -q exits 0 on match, 1 on no match, with no output. The return code assertion is the signal.
The same applies to wc -l, grep -c, or any count-based check used as an existence gate.
Run SSH command already wraps the remote command in a quoted Gherkin string
and sends it through a remote Bash shell. Put the Python expression in Bash
ANSI-C single quotes ($'...') and use double quotes inside the Python code
so shell word-splitting cannot break expressions containing parentheses or
dictionary literals.
* Run SSH command: "bootc status --json | python3 -c $'import sys,json; d=json.load(sys.stdin); print(d.get(\"status\",{}))'"
The $'...' ANSI-C quote is intentional: the shared SSH step wraps commands
with bash -c, which consumes the escaped quotes before Python receives them.
Audit every python3 -c added to an SSH command for this pattern. Avoid
nested escaped double quotes around the entire Python expression; those
escapes can survive into the remote shell and expose Python punctuation to
shell parsing.
gsettings get returns GVariant text such as uint32 1. assert "1" in output
also matches uint32 10, uint32 11, and uint32 100, so a failed state change
reads as a pass. Parse the payload and compare it exactly:
_UINT32_RE = re.compile(r"^(?:uint32\s+)?(\d+)$")
def _parse_index(output: str) -> int | None:
match = _UINT32_RE.match(output.strip())
return int(match.group(1)) if match else None
assert _parse_index(output) == 1, f"Current input source is not 1: {output}"
An unparseable value must fail, never pass by default.
A cleanup helper that saves state and restores it later is normally guarded by a
_restored flag so the explicit step and the context.add_cleanup hook do not
run twice. Set that flag only after every restore command actually returned 0:
failures = []
for key in ("sources", "current"):
output, rc = _run_in_vm_checked(f"gsettings set {schema} {key} {shlex.quote(value)}")
if rc != 0:
failures.append(f"{key} (rc={rc}): {output}")
if failures:
raise AssertionError("Failed to restore ...: " + "; ".join(failures))
state["_restored"] = True
Latching the flag on a failed command turns the cleanup hook into a no-op: the retry never happens and the mutated state leaks into every later scenario in the run. Surface the failure instead of swallowing it.
ujust recipe coverageDakota's interactive recipes shadow fzf/gum/gh on PATH; mocks must answer each
prompt distinctly and assert on the invocation, not merely that the tool ran
(reference).
Keep SSH-based ujust recipe checks in tests/common/features/common_ujust.feature.
Prefer assertions against the wrapper's own output, not the underlying tool's raw
output — for example ujust bios-info prints Manufacturer: / Release Date:
labels itself, so checking for raw dmidecode keys like Vendor is brittle.
If a recipe is gated by gum choose, pkexec, or package-install side effects,
prefer a non-interactive entry point in the recipe itself. Where the image has
not shipped one yet, land the coverage behind a @requires_* skip tag so it
activates when the contract lands, instead of leaving the scenario @pending
forever. ujust toggle-updates flips uupd.timer or
rpm-ostreed-automatic.timer (not ublue-update.timer) and is covered via its
non-interactive ACTION=enable|disable|cancel argument (see the reference
below).
Two recipes are worked out in detail in
references/ujust-noninteractive.md:
toggle-updates gained a non-interactive ACTION entry point in
projectbluefin/common and is covered by a @requires_toggle_action scenario
(projectbluefin/testsuite#499), while toggle-devmode has one via
bctl devmode --enable/--disable (projectbluefin/testsuite#500).
Do not simulate uupd's battery or metered-network suppression in testsuite
scenarios. uupd reads org.freedesktop.UPower and
org.freedesktop.NetworkManager system-bus properties, and the current image
and lab contracts do not provide a safe, isolated way to override those
properties and restore the host state. The /sys/class/power_supply files
and GNOME proxy settings are not the interfaces uupd checks.
Until the image or lab supplies a supported simulation hook, keep the existing
uupd presence/timer health check and document this as a coverage gap. A future
implementation must use the upstream uupd config path (/etc/uupd/config.json)
and keys documented by ublue-os/uupd, and must define how the UPower and
NetworkManager state is injected without changing the host's persistent state.
Use xdg-mime query default <mime-type> to verify handler registration without launching apps. Assert against known .desktop file names or an allow-set:
DOCUMENT_VIEWERS = {"org.gnome.Papers.desktop", "evince.desktop"}
actual = subprocess.run(["xdg-mime", "query", "default", "application/pdf"],
capture_output=True, text=True).stdout.strip()
assert actual in DOCUMENT_VIEWERS
This validates the MIME database end-to-end (xdg-mime, .desktop file registration, mimeapps.list) without window management flake.
For lightweight smoke-suite CLI assertions (for example gsettings, pgrep, or journalctl checks), prefer qecore's built-in command capture steps in the .feature file instead of adding wrappers:
For MIME-handler coverage, prefer direct subprocess.run(["xdg-mime", "query", "default", mime]) helpers in tests/smoke/features/steps/steps.py and assert the resolved .desktop file (or an allowed viewer set) instead of launching the app with xdg-open. This keeps smoke checks local to the VM and avoids window-management flake while still validating Bluefin's handler registration end to end.
* Run and save command output: "gsettings get org.gnome.desktop.a11y.keyboard enable"
* Return code of last command output "is" "0"
* Last command output "contains" "true"
Do not invent alternate phrases like Run command: / Command output contains ... in smoke features unless you are also intentionally adding matching step definitions.
When a smoke assertion needs an allow-set result (for example systemctl is-enabled
returning either enabled or static), keep using the built-in qecore command
steps and encode the allow-set in the shell command itself:
* Run and save command output: "systemctl is-enabled uupd.timer | grep -E '^(enabled|static)$'"
* Return code of last command output "is" "0"
This avoids adding a one-off step definition just to express "one of these two values is acceptable".
For smoke steps that must run several host commands, reuse the suite's canonical
_run_host helper with from steps.steps import _run_host. It handles local
qecore-headless execution and the runner-container SSH fallback. Do not duplicate
SSH argument construction or container detection in a new smoke step module.
Flatseal (com.github.tchx84.Flatseal) is only a front end over flatpak override
and the portal permission store. Cover per-app permission behaviour with the CLI —
it needs no desktop session and no AT-SPI:
| What Flatseal shows | CLI assertion surface |
|---|---|
| Per-app toggles the user has changed | flatpak override --user --show <app> |
| Effective manifest permissions | flatpak info --show-permissions <app> |
| Portal grants (documents, notifications, background) | flatpak permissions [<table>] |
Two properties make these scenarios survivable in CI, where
flatpak-preinstall.service is masked and /var/lib/flatpak is not seeded
(the reason tests/smoke/features/flatpak_permissions.feature is quarantined):
flatpak override --user accepts an application ID that is not installed.
Use a synthetic ID such as org.projectbluefin.TestsuitePermissionProbe so the
round-trip neither depends on nor clobbers real installed apps. Always finish the
scenario with Reset flatpak user overrides for ....Every installed flatpak app exposes a parsable permission set is @pending on
#706. More traps: references/flatpak-permissions.md.flatpak override --show emits a keyfile, not flag syntax:
[Context]
sockets=!wayland;
Parse it (parse_flatpak_context in
tests/software/features/steps/flatpak_permissions_steps.py): split on the first
= and compare the key — matching bare key names against filesystems=home;
never matches and passes falsely.
tests/software/features/environment.py calls populate_ssh_context(context) in
before_all (#718), so the shared tests/shared/ssh_steps.py steps work there.
@flatpak_cli marks image-agnostic software scenariostests/software/features/environment.py skips any @software scenario when Bazaar
(io.github.kolunmi.Bazaar) is absent — unless the scenario also carries
@flatpak_cli. Tag CLI-only, image-agnostic software scenarios with @flatpak_cli
so they still run on gnomeos and other non-Bluefin images.
@requires_cached_image gates scenarios on a pre-pulled OCI imageA scenario must never pull the container image it needs — a cold
distrobox create pulls inside the scenario and eats the CI timeout (#501).
Tag it @requires_cached_image; skip_when_image_not_cached() in
tests/shared/image_cache.py, called from the suite's before_scenario right
after skip_quarantine, reads the image refs out of the scenario's own step
text, probes each with podman image exists on the DUT, and skips while any is
absent. The scenario then activates on its own once the image is cached.
It is a runtime capability gate like @requires_bctl, not a non-runnable
tag: keep it out of _SKIP_TAGS / NON_RUNNABLE_TAGS / BEHAVE_TAG_ARGS, and
never pair it with @pending or @future — skip_quarantine returns first and
the gate goes inert. See
the cached-image gate reference.
Use @future when the step implementation isn't ready yet:
@future
Scenario: Hardware watchdog is active
Given the VM has TPM 2.0
...
Remove @future when all three are true:
Find remaining stubs:
just list-stubs
# or
grep -r "@future" tests/*/features/*.feature
Use a plain @retry tag on scenarios whose common failure mode is
infrastructure timing (GNOME session startup, AT-SPI render races, slow app
launch in QEMU). tests/shared/behave_retry.py only re-runs failing rerun
entries whose effective tags include retry; untagged failures fail the job
immediately after the first pass.
The retry budget comes from BEHAVE_RETRIES / --retries (default: 2), so
@retry means "eligible for the normal retry loop" rather than a per-scenario
count override.
tests/
smoke/ # GUI smoke — behave + dogtail
developer/ # developer tools
software/ # software installation
flatcar/ # Flatcar compatibility
lifecycle/ # bootc upgrade/rollback
security/ # security posture
dx/ # DX variant
nvidia/ # GPU tests
hardware/ # hardware feature detection
vanilla-gnome/ # upstream GNOME parity
shared/
ssh_steps.py # canonical SSH step library
Each suite has:
<suite>/
features/
*.feature
environment.py
steps/
steps.py (+ additional step files)
Each suite loads only its own steps/*.py files plus qecore.common_steps. A step defined in tests/software/features/steps/steps.py is not available in tests/developer/features/steps/steps.py even if both import qecore.
Rule: When the audit agent (or any agent) moves a step phrase from a shared/smoke context into a suite-specific file, verify that every .feature file using that phrase is in the same suite. If multiple suites use the phrase, define it in each suite's steps.py.
Lesson surfaced 2026-05-30: No journal entries match "{pattern}" was added to software/steps.py but ptyxis.feature (developer suite) also used it — causing UndefinedStep at runtime.
Isolation cuts the other way too: an environment.py hook that imports a module containing @step decorators registers those phrases into the suite it runs in. tests/shared/ssh_steps.py collides with the DX suite's own SSH command return code is "{code}", so hooks that need to run a command on the DUT resolve connection details from tests/shared/ssh_config.py and call subprocess directly rather than importing the step library (#501).
behave --format rerun on 1.3.x adds header comments like:
# -- RERUN: 7 failing scenarios during last test run.
tests/smoke/features/foo.feature:5
tests/shared/behave_retry.py must filter out comment or non-.feature[:line]
entries before retrying. Passing those lines back to behave causes:
ConfigError: No steps directory in '/.../# -- RERUN: 7 failing scenarios ...'
Last command output stripped "is" vs multiline outputstripped "is" "<value>" strips whitespace from the entire captured output and checks equality. This only works correctly for single-line command output (e.g. grep -c, echo X).
For commands that produce multiline output (e.g. flatpak install ... 2>&1; echo rc:$?), use Last command output contains "rc:0" instead.
# WRONG — fails when flatpak install produces install progress lines
* Last command output stripped "is" "rc:0"
# CORRECT — substring check works with multiline output
* Last command output contains "rc:0"
gsettings get vs dconf readIn the SSH-driven common suite, validate Bluefin desktop identity overrides with the API that matches the schema type:
gsettings get for regular schemas shipped via zz0-bluefin-modifications.gschema.override (for example org.gnome.desktop.interface accent-color or org.gnome.desktop.app-folders folder-children).dconf read for relocatable schemas and extensions without XML schemas (for example custom media-key keybindings under /org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/, Search Light under /org/gnome/shell/extensions/search-light/, and Ptyxis profile palette keys under /org/gnome/Ptyxis/Profiles/<uuid>/).This keeps common-suite assertions aligned with how Bluefin actually ships those defaults in projectbluefin/common.
When a scenario fails in QEMU CI due to a known environment limitation (not a code bug), tag it @quarantine and file a tracking issue:
@quarantine
Scenario: Flathub remote is configured
...
The CI runs behave with --tags ~quarantine, so quarantined scenarios are skipped and do not block the gate.
| Situation | Action |
|---|---|
| Scenario fails because a first-boot service is masked in CI | Quarantine + image-side issue |
| Scenario fails due to GNOME 50 API change | Quarantine + investigate new API |
| Scenario fails due to a real regression in the image | Do NOT quarantine — it's a gate hit |
| Scenario fails intermittently (flaky network, timing) | Add @retry tag instead |
smoke-b: screen lock (#528), PDF/PNG/video MIME defaults (#529)
common-a/b: 13 scenarios in dconf, flatpak, immutable, polkit (#531)
Behave's Background: runs before every scenario in the feature, including scenarios that don't
need the app to be open (e.g. coredump checks, version assertions, cleanup verifications).
Anti-pattern:
Feature: GNOME Settings smoke tests
Background:
* Settings window is accessible # ← runs before EVERY scenario
Scenario: Settings closes cleanly via Ctrl+Q
* Key combo: "<Ctrl><Q>" with uinput
* Settings is no longer running # ← explicitly kills Settings
Scenario: No gnome-control-center coredump after session start
* No coredump entries exist for "gnome-control-center"
# ^ Background runs first: tries to find Settings after it was just killed → FAILS
Fix options (prefer first):
Background:
* Launch Settings via command
* Settings window is accessible
Rule: A feature Background should only assert state that is valid for ALL scenarios in that feature. If any scenario tears down that state, either restructure the teardown or move the non-dependent scenarios to a separate feature.
| Rationalization | Reality |
|---|---|
| "I'll just add a tiny local SSH helper." | Shared SSH phrases already exist in tests/shared/ssh_steps.py; duplication creates drift and inconsistent assertions. |
| "This phrase is descriptive enough; collisions are unlikely." | Behave loads all step files in a suite together, and duplicate phrases fail at runtime with AmbiguousStep. |
| "Equality is stricter, so I'll use it for all command output." | ... stripped "is" only works for single-line output; multiline commands need substring assertions. |
| "A dry-run is overkill for a simple feature edit." | CI runs dry-run and will catch missing step definitions immediately; local dry-run is the cheap failure path. |
Run SSH command or other shared SSH assertions.feature file introduces a phrase with no matching @step decoratorgsettings vs dconf read)tests/** without a matching skill update@retry added to a scenario to mask a missing retry loop in the underlying _<app>_app() helperbehave --dry-run passes for the touched suite/behave/behave