ci

# Ginkgo in CI A CI invocation should maximize signal — surface flakes and spec pollution, catch mistakes that pass locally, and emit machine-readable artifacts — while collecting *every* failure in one run. This builds on the CLI (`ginkgo:running`) and report formats (`ginkgo:reporting`). Full rationale: <https://onsi.github.io/ginkgo/#recommended-continuous-integration-configuration>. ## The recommended flag set Invoke via `go run` so the CLI version always tracks the `github.com/onsi/ginkgo/v2` in your `go.mod` — no separately-installed binary to drift (→ `ginkgo:setup`). ```bash go run github.com/onsi/ginkgo/v2/ginkgo \ -r -p --randomize-all --randomize-suites \ --fail-on-pending --fail-on-empty --keep-going \ --cover --coverprofile=cover.profile --race --trace \ --json-report=report.json --output-dir=.ginkgo-report \ --timeout=TIMEOUT --poll-progress-after=Xs --poll-progress-interval=Ys ``` | Flag | Why | |---|---| | `-r` | recursively find and run every suite | | `-p` | run each suite in parallel (→ `ginkgo:parallelism`; set `--procs=N`/`--compilers=N` if CPU detection is wrong) | | `--randomize-all` / `--randomize-suites` | shuffle all specs and the suite order to surface spec pollution (→ `ginkgo:running`) | | `--fail-on-pending` | fail if any `Pending` specs were committed | | `--fail-on-empty` | fail if no specs ran (usually a malformed filter) | | `--keep-going` | don't stop at the first failed suite — collect all failures | | `--cover --coverprofile=cover.profile` | compute coverage into one merged profile (→ `ginkgo:reporting`) | | `--race` | run with the race detector (costs a fixed ~1s per suite — see below) | | `--trace` | full stack traces on failure (worth it without a local feedback loop) | | `--json-report=report.json` | structured results for diagnosis and downstream tools (→ `ginkgo:debugging-failures`) | | `--timeout=TIMEOUT` | cap the whole run (default 1h — often not enough) | | `--poll-progress-after`/`--poll-progress-interval` | emit progress reports for stuck specs (→ `ginkgo:timeouts-and-async`) | ## The exit-code safeguards — CI's real value Three flags turn "looks green" into "is actually trustworthy." They make CI fail on mistakes that pass silently on a developer's machine: - **`--fail-on-pending`** — a committed `Pending`/`PIt`/`XIt` becomes a CI failure, so dev-time placeholders can't rot into the suite. - **`--fail-on-empty`** — a typo'd `--label-filter` (or an over-aggressive `--skip`) that selects *zero* specs fails instead of falsely passing. - **Committed programmatic focus already fails CI for free.** A *passing* suite that contains `FIt`/`FDescribe`/`Focus` **exits non-zero** — Ginkgo's built-in guard against shipping focus. Don't suppress it; fix it with `ginkgo unfocus` (→ `ginkgo:filtering`). ## Collecting artifacts Point reports and profiles at one directory and upload it as a build artifact: - `--output-dir=.ginkgo-report` collects the JSON report (and, under `-r`, merges all suites into one `report.json`) plus coverage/profiles into one place. → `ginkgo:reporting` for `--keep-separate-reports` and the other formats. - `--junit-report=junit.xml` *additionally* if your CI system renders JUnit — but keep `--json-report` as the source of truth; JUnit loses Ginkgo metadata. - An agent or human then diagnoses failures straight from `report.json` with `jq` → `ginkgo:debugging-failures`. ## CI-friendly console output - **`--github-output`** — formats the console log for GitHub Actions readability (grouped, annotated). - **`--force-newlines`** — flush output line-by-line for CI systems that only flush on newline. - **`--no-color`** (or `GINKGO_NO_COLOR=TRUE`) — drop ANSI codes from logs that don't render them. ## The race detector's fixed per-suite cost `--race` adds about a second to **every** suite, no matter how few specs it has. ThreadSanitizer sleeps for one second at process exit (`atexit_sleep_ms`, its default) so that races reported by goroutines still running at exit aren't lost, and Ginkgo has to wait for the processes it spawned. Under `-r -p` that lands in the gap *between* suites, where it looks exactly like compilation. **The tell is that it's flat across package size** — a 2-spec package pays the same as a 2000-spec package. A build cost can't behave that way; a per-process cost must. If you're auditing a slow race-enabled run, measure with precompiled binaries (`ginkgo build -r`) so compilation is excluded by construction rather than by inference. Remove it by zeroing the sleep: ```bash GORACE=atexit_sleep_ms=0 go run github.com/onsi/ginkgo/v2/ginkgo -r -p --race ... ``` This slightly weakens detection at the very *end* of a suite — a reasonable trade for a suite whose races are found by its specs rather than at teardown, but not one to make blindly. Write the reason at the call site. ## Flakes and timeouts in CI - **Collect, don't bail.** `--keep-going` (across suites) plus Ginkgo's default keep-going *within* a suite gives you the full failure picture in one run. Avoid `--fail-fast` in CI — it truncates the report (→ `ginkgo:debugging-failures`). - **Don't paper over flakes.** `--flake-attempts=N` exists, but treat it as explicit, temporary tech debt rather than a standing CI flag — fix the root cause (→ `ginkgo:ordering-and-flakes`). - **Set `--timeout` deliberately** (default 1h) and add `--poll-progress-after`/`--poll-progress-interval` so a stuck spec emits a progress report (current node + goroutine stacks) instead of silently eating the budget. For long suites, `120s`/`30s` are reasonable; skip them for fast unit suites. If you precompile and run elsewhere, set `--source-root` so progress reports can show source. ## See also - The CLI surface and randomization details → `ginkgo:running` - Report formats, programmatic reporters, and profiling → `ginkgo:reporting` - Reading the JSON report after a failed CI run → `ginkgo:debugging-failures`