testing-and-ci

# Testing and CI ## Browser choice Use **headless Chrome** for browser E2E tests, screenshots, developer-tools network traces, and conformance visual evidence. Avoid Firefox unless the user explicitly requests it for a browser-specific investigation. Prefer a pinned Chrome for Testing distribution for reproducible external workflows; keep its binary, driver, profiles, caches and artifacts under this repository's `tmp/`. Preserve real browser interaction and TLS validation. Do not set `acceptInsecureCerts`, ignore-certificate flags, disable web security, rewrite Origin headers, or fabricate successful callbacks to make a conformance run pass. ## Overview Use this skill for caddy-security test selection, fixture maintenance, and CI reproduction. Select tests from the changed behavior for both development and final validation. Do not run the full suite as a routine completion step. Use filtered Makefile targets when reports are needed; reserve full-suite commands for explicitly requested full-suite, CI-reproduction, or release workflows. The Go module is rooted at the repository top level. Most tests live in the root `security` package and cover Caddyfile parsing, Caddyfile adaptation, and runtime authcrunch config resolution. Bundled backend companions live in `plugins/<category>-<backend>`. Run `go test -mod=readonly -race -count=1 ./plugins/... ./pkg/plugin` for their category-local parser, factory, real backend and ownership tests, plus direct OAuth storage metadata validation. Keep public tests in external test packages; private runtime-state tests stay with their category. The standalone inventory in `plugins/consumer_test.go` intentionally does not import the parent app. Category-local adapt integration tests register the app explicitly. Select the affected root dispatch, native alias, storage/lifecycle and actual Caddy E2E parents as well. See the [companion test inventory](references/test-surfaces.md#bundled-companion-packages). Collect companion statement coverage explicitly when changing these packages: `make test TEST='^Test' TEST_DIR='./plugins/... ./pkg/plugin' COVERAGE_DIR='.coverage/plugin-package-tests'`. Inspect `go tool cover -func=.coverage/plugin-package-tests/coverage.out` and uncovered branches; a root-only profile does not measure extracted companions. Keep category contracts and their executable tests in separate `pkg/plugin` files. Interface-only contracts do not need existence assertions. Each companion owns `caddyfile_adapt_test.go` and fixtures directly under its `testdata/` directory. Use the shared runner in `internal/caddyfiletest` to adapt complete Caddyfiles through the registered Caddy adapter. Pair each input with exactly one semantic `.json` expectation or exact `.error` expectation; optional `.env` entries are restored after each case. Select these tests with `make test TEST='^TestCaddyfileAdapt|^TestCompanionModules' TEST_DIR='./plugins/...'`. Keep root app adaptation and runtime-resolution fixtures in their existing location. These companion fixtures qualify configuration output and errors; they do not provision backends or replace runtime E2E coverage. Follow the [repository scope](../coding-directives/SKILL.md#repository-scope). Create and run tests in this module only; never run sibling suites to validate work here or fix a failing test by editing the sibling. `TEST_DIR` and `QUICK_TEST_DIR` must select this module's packages, and `COVERAGE_DIR` must not point into another checkout. Existing local replacements are dependency inputs only. Keep compatibility fixtures and reports here and report required upstream test or implementation changes as separate work. ## Required Coverage for Code Changes When writing or changing code, ensure both unit tests and E2E tests exist and exercise the changed behavior. Add or amend tests where coverage is missing; existing tests count when they demonstrably verify that behavior. Run the relevant unit and E2E tests before considering the code change complete. Unit tests should check focused behavior and meaningful failure cases. E2E tests should exercise the affected user-visible flow through the assembled system. For authentication, authorization, and lifecycle changes, use actual Caddy provisioning, routes, and HTTP requests as appropriate. Bound network, process, and worker completion; clean up test-owned resources. Parser/adapt tests and sibling go-authcrunch tests do not replace this repository's E2E coverage. Report missing coverage or blocked validation explicitly. Use `lifecycleAddress` for disposable Caddy listener addresses. It checks both TCP and UDP availability because Caddy's default HTTPS server also starts a QUIC listener; a TCP-only ephemeral-port check can select an occupied UDP port. The occupied-QUIC-port regression also verifies release of the rejected TCP reservation. Reservations are released before Caddy takes ownership. Every Caddyfile directive change also requires new or amended adaptation test cases in `testdata/caddyfile_adapt/`, even when the resulting JSON shape is unchanged. Register new cases in `TestCaddyfileAdaptAuthenticationToJSON` in `caddyfile_adapt_test.go` so the fixture is exercised. This is additional to unit and E2E coverage; use the fixture mechanics below. Documentation/skill-only edits use metadata, link, and source checks from [skill-authoring](../skill-authoring/SKILL.md); they do not require new runtime tests. ## Select Tests From the Change Before running tests, inspect the diff, changed functions and their consumers. Use `rg` and the relevant part of [test surfaces](references/test-surfaces.md) to identify the smallest sufficient set of existing or new tests. Feature test inventories describe available coverage; they are not lists to run in full for every change. - Select unit tests for the changed behavior and meaningful failure paths, plus the E2E parent tests that exercise the affected user flow. Include adjacent consumers only when they share the changed contract or helper. - For parser changes, include the affected adaptation subtests. Add runtime resolution subtests when resolved output or replacement behavior changes. A fixture-only correction need not run every adaptation or resolution case. - For test-helper changes, run its focused regression tests and the E2E parents that consume it. Select the parent that launches an isolated process; running an environment-gated process helper directly can report only a skip. - For dependency changes, inspect changed upstream contracts and select their host integration tests. A version bump alone does not require every suite. - For documentation/skill-only edits, use metadata, link and source checks. State which tests were selected and why they cover the change before execution. Choose the affected package (`.` for root-package tests), and use `-run` with specific test names or subtest selectors. Use `-count=1` for fresh evidence and `-race` where concurrency, shared state or lifecycle behavior needs it. Filtered Make workflows already enable both. Stop when the selected checks pass. Broaden or repeat only after new edits, failures, or an identified coverage gap; explain which additional behavior the new selection verifies. Do not automatically follow focused checks with `go test ./...`, unfiltered `make test`/`make qtest`, or `make ci-check`. In the final report, name the checks actually run and their scope; a focused pass does not establish full-suite success. ## Syntax and Example Audits Follow [Syntax maintenance](../configuration/references/syntax-maintenance.md) when parser grammar or a dependency changes. Inventory standalone Caddyfiles, Go syntax comments, skill examples, and inline positive/negative tests. Adapt runnable examples with a binary built from the selected dependencies. Wrap fragments in their actual global/portal/policy/site scope; do not run syntax catalogues or intentionally invalid examples as complete configurations. Adaptation verifies only the validation reached by that parser. Raw crypto, messaging, registration, ACL, and other deferred settings also need focused resolution/validation checks when their examples change. Do not provision a real deployment, contact an external provider, or change machine trust merely to audit syntax. Preserve intentional failure fixtures and legacy alias tests. Report external-module requirements and runtime checks separately from adapt success. Comment-only edits do not change parser behavior; use source review, formatting, and relevant existing tests instead of adding tests that mirror prose. ## Command Selection Use direct Go tests with explicit selectors and packages. For example, a change to the SQLite COMMIT uncertainty fixture selects its regression and client E2E: ```bash go test -mod=readonly -race -count=1 -run '^(TestSQLiteRefreshCommitFault|TestCaddySQLiteRefreshUncertaintyE2E)$' . ``` Use filtered `make test` or `make qtest` for the repository report lifecycle. Both need an explicit `TEST` selector: `qtest` narrows the default package to `.` but still runs every test in that package when `TEST` is omitted. Set `TEST_DIR` or `QUICK_TEST_DIR` to the affected packages. `go.mod` pins `github.com/greenpau/tested`, invoked as `go tool tested`; it owns `-json`, `-coverprofile`, child-process status, and coherent reports. Do not reintroduce `go test | tee`, log-grep success detection, richgo, tparse, or go-test-report. ```bash make test TEST='^TestSQLiteRefreshCommitFault|^TestCaddySQLiteRefreshUncertaintyE2E' TEST_DIR='.' make qtest TEST='^TestSQLiteRefreshCommitFault|^TestCaddySQLiteRefreshUncertaintyE2E' QUICK_TEST_DIR='.' make run-reports ``` The two test commands are alternatives for the same selection; do not run both without a reason. Run `make test-automation` when the changed automation needs its fixtures. Unfiltered `go test ./...`, `make test` and `make ci-check` belong to the explicit full-workflow scope described above and under CI Workflow. Lifecycle runs use `-mod=readonly -race -count=1 -p 1 -parallel 2 -timeout 60m -v`. The macOS/Linux resource guard bounds the whole process tree, including compilers, browser/CLI children and report rendering. Read [test resource controls](../scripts-and-automation/references/test-resources.md) when changing defaults or investigating an interrupted run. The default wall limit is 4,200 seconds, allowing compilation/report time around the 60-minute package limit. Keep live tested output enabled; the guard also prints progress every ten seconds. `TEST` is a regex (default `.`), `TEST_DIR` accepts package patterns (default `./...`), and `TEST_TIMEOUT` overrides the quoted per-package limit. Make expands dollar signs in command-line variables even inside shell single quotes. For example, `TEST='^(TestA|TestB)$|^TestC'` lets Make consume `$|` as an automatic variable, changing the selector and potentially omitting tests while the process still succeeds. Prefer prefix selectors such as `TEST='^TestA|^TestB|^TestC'` when sufficient. If an end anchor is necessary, escape its dollar for Make: `TEST='^(TestA|TestB)$$|^TestC'`. Direct `go test -run` uses the ordinary single `$`; the extra escape is for Make. Inspect the effective `-run` argument in the bundle's `run.json` command and the selected test names in `test_output.jsonl` before claiming the intended tests passed. A zero exit status or aggregate test count alone does not prove selection. `MINIMUM_COVERAGE` defaults to 1 percent as a nonzero-profile check, matching go-authcrunch; it is not a substantial coverage target. Go's timeout covers the entire package, including all sequential E2E parent tests. The Caddy journeys also have their own shorter child-process deadlines. If CI times out, inspect the captured test events and active test duration to distinguish an exhausted package budget from a stuck individual journey. Keep the job budget larger than the package budget so setup, builds and report upload can finish; hosted shard jobs allow 85 minutes around the 60-minute Go package limit and a separate guarded discovery pass. Reports land in `.coverage`. `make qtest` defaults to the root package (`.`) with reports in `.coverage/quick`; override `QUICK_TEST_DIR` and `TEST` for another scope. Use `COVERAGE_DIR` for separate evidence bundles; guarded runs in the same checkout cannot overlap, even with different output directories. Let tested refresh its managed files without deleting other bundles or investigation notes. `make run-reports` regenerates presentations from recorded evidence and preserves failures; `make coverage` aliases it without rerunning tests. Use `make build` when validation needs `bin/authcrunch` or `bin/caddy-authenticator`; it builds both and prints their versions. Formatting is separate: `make fmtcfg` formats fixtures under `testdata/caddyfile_adapt` and `assets/config`. Builds/tests do not rewrite licenses, version files, module manifests, or Caddyfiles. `make dep` downloads/verifies pinned dependencies and resolves tested; it may need network access but does not install global tools. ## Test Surfaces Read [test surfaces and qualification evidence](references/test-surfaces.md) when selecting existing unit/E2E tests, extending a feature suite, or changing coverage collection. It maps conditional authentication, CodeQL, subprocesses, logging, persistent state, operator examples, OIDC, refresh, native/CLI clients, local identity, authorization paths, lifecycle, and Caddyfile fixtures to their actual tests and limits. For interactions among login, refresh, OIDC, upstream OAuth, authorization, edge metadata and reload, read [composition qualification](references/composition-qualification.md). The feature owner's acceptance contract determines what must be exercised; a list of parser fixtures alone never proves a user flow works. ## Subprocess coverage Root-package E2E helpers run the same instrumented test executable. The [subprocess coverage contract](references/test-surfaces.md#subprocess-coverage) owns collection before parent completion, isolated child directories, counter publication, killed-process limits, and the regression workflow. Read it when adding a subprocess or changing report collection. ## Adding Coverage When adding or changing a Caddyfile directive, add focused parser coverage in the nearest `caddyfile_*_test.go` file. Include both the successful config shape and a malformed input when the parser has a meaningful error path. For a Caddyfile directive change, add or amend adaptation cases in `testdata/caddyfile_adapt/`: `<prefix>.Caddyfile`, `<prefix>.json`, and optionally `<prefix>.env`, and update the test case registration as needed. If runtime defaults, replacements, credentials, UI, OAuth, registration, or cookie behavior changes after adaptation, also add or update `<prefix>_resolved.json` and include the prefix in `TestResolveRuntimeAppConfig`. After fixture edits, select the affected subtests by their `t.Run` names, which can differ from fixture filenames. For example, changes to named refresh-store adaptation and runtime resolution use these selections: ```bash go test -mod=readonly -count=1 -run '^TestCaddyfileAdaptAuthenticationToJSON$/named_SQLite_refresh_store' . go test -mod=readonly -count=1 -run '^TestResolveRuntimeAppConfig$/named_refresh_storage' . ``` Add the relevant parser unit and runtime E2E tests if implementation changed; passing the affected fixtures does not require an automatic full-suite run. If `*_tmp_input.json` or `*_tmp_output.json` files remain after a failing runtime resolution test, inspect them, fix the fixture or implementation, and remove the generated temp files before finishing unless the user explicitly wants debug artifacts kept. ## CI Workflow `.github/workflows/build.yml` runs on pushes/PRs to `main`, manual dispatch, and reusable workflow calls. A planning job resolves one artifact identity. Repository quality gates and shared build preparation run alongside one another, then 44 Go shards run in separate checkouts: 43 disjoint root selections, including 24 balanced runtime groups, and one containing every other package. Preparation derives browser prerequisites from the actual inventory and shares compiled builds to reduce repeated setup. A quality or preparation failure skips the whole matrix. Local `make ci-check` remains sequential. The final `Tests and coverage` job preserves the existing branch-protection and release check. It requires every dependency to succeed, verifies each discovered root test completed exactly once, validates the report evidence, and merges atomic coverage counters without counting root statements multiple times. Failed or unexpectedly skipped jobs cannot produce a successful gate. Read the [parallel CI contract](references/parallel-ci.md) for shard ownership, new-test fallback, cache keys, browser prerequisites, failure artifacts, coverage integrity and automation regression coverage. The workflow retains the existing release-main-push skip; the reusable tag validation still runs all gates before GoReleaser. Actions use immutable revisions and read-only contents permission. Hosted resource overrides and the 85-minute shard budget leave local defaults unchanged; see [test resource controls](../scripts-and-automation/references/test-resources.md). When the user explicitly requests local CI reproduction, use: ```bash make dep make ci-check ``` Release CI, tag selection, artifact naming, packaging checks, and publication scope belong to [release-and-versioning](../release-and-versioning/SKILL.md). The release workflow calls this reusable gate before GoReleaser. `make ci-check` serializes version validation, Python automation fixtures, the full Go report lifecycle, and the binary build, even under `make -j`. The full Go suite includes real-browser refresh E2E through Caddy. It serves the selected go-authcrunch embedded UI rather than building or running tests in the sibling checkout. The existing `make linter` remains a placeholder and is not a gate. When changing tested or its invocation, run `make test-automation`. It exercises real Make/tested processes in disposable repositories: filtering, full/quick/ custom bundle isolation, assertion failures, compile failures, short timeouts, and failed offline reports. A live-log handshake proves output reaches Make before the selected Go test completes. Guard unit and Make E2E fixtures cover accounting, process cleanup, resource refusal, cancellation, lock contention, nonblocking output and Linux process-exit races. The subprocess fixture retains exact descendant coverage through guarded test and report invocations. Version fixtures exercise the public artifact command and validated `GITHUB_OUTPUT` values without publishing remotely. Archive-checker unit fixtures cover missing targets, checksum failures, mixed binaries, incorrect documents and Unix executable permissions. For GoReleaser packaging changes, also run a real snapshot release and the archive checker per the [packaging workflow](../release-and-versioning/references/ci-and-packaging.md#toolchain-and-packaging-checks); fixture tests cannot establish cross-compilation or actual archive assembly. The CLA workflow may update `assets/cla/signatures.json` through GitHub automation. Do not edit CLA signatures or consent files unless the user asks. ## Generated Artifacts Go's `./...` discovery does not honor `.gitignore`. Name temporary Go helpers with a leading underscore, or put them in an underscore-prefixed directory, so ignored working files do not become extra test/coverage packages. Confirm the package list with `go list -mod=readonly ./...`. Keep source files in place until both the covered test run and report generation finish: the coverage renderer still reads them after tests exit. Preserve a failed bundle before rerunning; do not remove source files or edit coverage profiles to repair it. Treat these as generated outputs unless the user explicitly asks to preserve or commit them: ```text bin/authcrunch bin/caddy-authenticator .coverage/index.html .coverage/summary.json .coverage/junit.xml .coverage/coverage.html .coverage/coverage.out .coverage/test_output.jsonl .coverage/test_output.html .coverage/stderr.log .coverage/run.json .coverage/manifest.json .coverage/resource-usage.json .coverage/resource-report-usage.json .coverage/test-resource.lock testdata/caddyfile_adapt/*_tmp_input.json testdata/caddyfile_adapt/*_tmp_output.json ``` The manifest is published last for a coherent generation; a failed run can leave only partial evidence. Inspect `run.json`, `stderr.log`, and `test_output.jsonl` before rerunning so failures are not overwritten without review. Also inspect `resource-usage.json` for budget or monitoring failures. A running or aborted guard record prevents offline reporting from accepting older tested evidence. Preserve interrupted bundles and rerun into a fresh directory. Test output and coverage sources are unredacted; use synthetic fixtures. Formatted Caddyfiles and JSON fixtures can be intentional source changes. Review the diff after explicit format or fixture updates. Skill-only edits use [skill-authoring](../skill-authoring/SKILL.md) and the default skill-creator validator instead of running the Go suite. ## Acceptance criteria - A code change is supported by unit and E2E evidence for its observable behavior; parser-only success cannot stand in for authentication, authorization, or restart. - Selected tests are justified by the diff and affected contracts. Passing them completes routine validation without a full-suite run; the report identifies the actual selection and any uncovered or blocked behavior. - The relevant repo-local skills reflect the final code and validation evidence, following [keeping skills current](../skill-authoring/SKILL.md#keep-skills-current-after-code-changes). Review affected examples, test references, and acceptance scenarios before calling the code change complete. - A failing test or scanner leaves its original status and diagnostic evidence visible. Offline report regeneration cannot convert a failure into a pass. - A skill-only edit runs metadata, link, routing, and source checks. It does not trigger runtime, release, or conformance workflows merely to validate prose.