testing-conventions

# Testing conventions (xberg-io/actions) Every action in this repo is a script plus a thin `action.yml`. The scripts are the product; the tests are how we know a change to one does not silently break a consumer's pipeline. ## Which framework, and where | Language | Framework | Location | Task | |---|---|---|---| | Shell `*.sh` | Bats | `<action>/tests/<name>.bats` | `task test:bats` | | Python `*.py` | pytest | `tests/test_<action_name>.py` (repo root) | `task test:unit` | | PowerShell `*.ps1` | Pester | `<action>/tests/<name>.Tests.ps1` | `task test:pester` | `task test` runs all three. Adding a script means adding its tests in the same change. Python suites live in the root `tests/` rather than beside the action because they share `tests/conftest.py` fixtures (`tmp_path_with_files`, `github_output`). Bats and Pester suites live beside their action, matching how the runner resolves them. ## The failure this repo actually had Read this before writing a test that touches a tool. Several Bats suites set `PATH="$STUB_BIN:/usr/bin:/bin"` and treated that as isolation. The scripts under test choose a branch by probing `command -v gh` / `gpg2` / `dotnet`. `ubuntu-latest` ships those in `/usr/bin`; macOS does not. So on CI the probe succeeded, the script took the already-installed branch, and **the branch under test never executed**. The suite reported 118/118 passing on a dev machine while two shipped actions were broken on every runner: - `setup-zig` used `mapfile`, a bash 4 builtin. macOS ships bash 3.2 as `/bin/bash`, which `#!/usr/bin/env bash` resolves to on a macOS runner. It died with `mapfile: command not found` and then read the resolved URL and version out of an empty array. Broken for every macOS consumer. - `build-swift-package` tried BSD `stat -f '%m'` before GNU `stat -c '%Y'`. On Linux, GNU `stat` does not reject `-f` — it reads it as `--file-system` and prints filesystem details starting `File: ...`, which reached an arithmetic comparison and aborted under `set -u`. Broken for every Linux consumer. Both were real product bugs the suite was correctly reporting; they had been dismissed as "flaky platform-specific tests". ## Isolating from the host **Exclude, do not allow-list.** Mirror the host's command directories into a private directory *minus* the one command whose absence is the point, then assert the mirror really lacks it so a leak fails loudly instead of quietly restoring the host's copy. An allow-list ("link in the utilities this script needs") guesses at an implementation detail and breaks on the first indirect dependency: GNU `tar -xzf` shells out to `gzip`, BSD `tar` does not, so a curated bin that omits `gzip` fails on Linux and passes on macOS. ```bash # see ensure-gh/tests/unix.bats for the working version expose_system_commands() { ... } # mirror host bins, minus the probed command ``` In pytest, `monkeypatch` every `shutil.which`, every `subprocess.run`, and every environment variable the script reads; prove it by running the suite with `env PATH=/nonexistent`. In Pester, `Mock` the probe (`Mock Get-Command`, `Mock Invoke-WebRequest`) rather than touching the real `PATH`. ## Portability constraints for scripts - **bash 3.2 is the floor.** No `mapfile`/`readarray`, no `${var^^}`, no associative arrays. Derive with `tr` instead of `^^`. - **GNU form first, BSD fallback second.** `stat -c '%Y' || stat -f '%m'`. GNU-first fails cleanly on BSD; BSD-first silently succeeds on GNU and emits garbage. - Verify with `bash --version` on a macOS runner, not locally, unless your local `/bin/bash` is genuinely 3.2. ## Proving a test discriminates A green run is not evidence. Before trusting a test, break what it covers and watch it fail: ```bash # python: mutate the script, run, restore, run # bats: break one assertion, confirm red, restore ``` Mutation-test the whole set when you add a suite. Two recent examples of why: one pytest suite had a test that still passed when the guard it covered was deleted, because control fell through to a different guard emitting superset stderr — fixed by asserting stderr exactly. A Bats fix was read as "unchanged" because the same test number kept failing, when it had actually moved from failing at line 54 to line 74 for a different reason. ## Honesty about coverage Pester suites run on `windows-latest` in CI. Locally on macOS or Linux, cover the portable logic (parsing, URL and filename construction, arch mapping, error paths) and mark genuinely Windows-dependent behaviour `-Skip` with a stated reason. A skipped test that says why is honest. A test that pretends to exercise Windows behaviour on macOS is not, and it will be believed. When a CI-only failure shows an assertion and no reason, the runner is not printing test output: Bats needs `--print-output-on-failure`, Pester needs `-Output Detailed`. Both are set in the workflows; keep them.