debug-failures

# Debugging Biloba failures Reading artifacts after a spec failed. To *prevent* flakes → `flaky-specs`. Docs: <https://onsi.github.io/biloba/#failure-artifacts-humans-ci-and-agents>. Sibling skills are named here without a prefix; invoke one with the same plugin prefix you loaded this skill under. ## Zero config: what you already get Biloba detects the environment in `ConnectToChrome`. "Automation" = `CI` is set **or** an AI coding agent is detected (`CLAUDECODE`/`AI_AGENT`/Cursor/Gemini CLI/Codex/…). | | Interactive human | CI **or** AI agent | |---|---|---| | Failure screenshot | inline in the terminal | written to `./biloba-screenshots` | | DOM outline on failure | no | yes | | Inline image blob | yes (if the terminal supports it) | no | So an agent or CI run needs nothing: `ginkgo -r -p`, then read the outline and the screenshot files. `BILOBA_SCREENSHOTS_DIR=./artifacts` points the directory elsewhere. ## Read the artifacts in this order 1. **Console errors** — any `console.error`/`console.assert` before the failure, replayed under "Console errors logged before this failure" at the **top** of the failure block. On a JS crash (a React error boundary) this is the root cause. 2. **`⚠` diagnostic notes** — they name the cause outright. 3. **Poll trajectory** — what the read you failed on did over its whole deadline (absent when the failure had no value read behind it). 4. **Visual diagnosis** — only on a failed `b.HaveScreenshot`: pixel counts plus the *shape* of the change, in words, before you open any image. 5. **Screenshot** — `Read` the printed PNG path. 6. **DOM outline** — "DOM Outline for: '<title>'": indented DOM, `<script>/<style>/<svg>` bodies pruned, whitespace collapsed, capped at 32 KB. Past `... [truncated]`? Raise the cap with **`BILOBA_OUTLINE_MAX`** — a byte count (`=131072`), or `0`/`off` for the whole DOM. ### Poll trajectory When an `Eventually` over a read that observes and compares a value (`b.HaveInnerText`/`b.HaveCount`/any value matcher, a geometry matcher, `b.EvaluateTo`, `b.GetJSValue`) times out, Biloba attaches the `(elapsed, value)` series **of that assertion**. Gomega's `Timed out … Expected <120>` shows only the final value; the shape is the diagnosis: | Shape | Means | Do | |---|---|---| | **flat** (one row, `held ×N`) | the product computed the value once and never reconciled | fix the product (`flaky-specs` §4) — a wider timeout won't help | | **monotone staircase** | latency; it nearly made it | widen the timeout | | **dip-then-rebound** | a late reflow shoved it back | settle layout before asserting | **No entry is a normal outcome, not a bug.** The series is claimed by the matcher Gomega asked for a failure message, so an entry always describes the read you failed on. Reads that passed, `b.Run` setup lines, and failures with no value read underneath (a `b.Click` whose selector never matched, a getter whose value was never there — that one gets the `AllowMissing` enrichment instead) simply produce nothing. Don't read a missing trajectory as a signal; go to the outline and the screenshot. To get a trajectory for an arbitrary expression, poll it with `b.EvaluateTo`/`b.GetJSValue` rather than wrapping `b.Run` in your own `Eventually` — Biloba only records reads it owns. On by default; `BilobaConfigPollTrajectory(false)` disables it (and the detached-node signal). **Don't disable it for speed.** Recording takes a lock and renders the value per poll sample, so it isn't free — but measured on a 1,558-spec suite with 2,470 call sites across the instrumented matchers (including gates asserting on whole paragraphs of prose), turning it off saved under a second across the whole run, inside that suite's own run-to-run spread. ### Visual diagnosis (a failed `b.HaveScreenshot`) → `visual-assertions` Two extra PNGs land in the screenshots dir — `<name>.actual.png` (what Biloba saw) and `<name>.diff.png` (the actual, washed out, differing pixels in magenta) — and the failure message says what moved: ``` screenshot "home-desktop" differs from baseline 38,160 of 1,017,600 pixels differ (3.75%), max channel delta 221 changed region: one box, (0,14)-(1272,44) [100% of the width, 4% of the height, at its top edge] unchanged: everything below y=44 baseline: /Users/you/app/biloba-baselines/home-desktop.png actual: /Users/you/app/biloba-screenshots/home-desktop.actual.png diff: /Users/you/app/biloba-screenshots/home-desktop.diff.png ``` The shape line (`one box`, `changed regions: N boxes`, `scattered`, `uniform shift`, a size change) and what to do about each → `visual-assertions`. When every differing pixel is off by only a few levels Biloba says `a rasterisation or compositing difference, not a content change` — nothing moved, so look for a shadow or gradient compositing into the capture. A **missing** baseline is a different failure: it says to re-run with `BILOBA_UPDATE_SCREENSHOTS=1`; never script your way past it. `Read` the `.diff.png` when the words aren't enough (inline images are off under an agent, so you get the path). **"never settled" is printed during an update run, not on a failure.** The run stays green, but the baseline it just wrote was captured from a page still changing and the next normal run will fail against it. Mask the moving region (or stop it moving) and re-run the update; re-running alone changes nothing. → `visual-assertions` **"Failed to clear the emulated prefers-color-scheme"** — a dropped `b.InColorSchemes` teardown. The override is target-level and survives navigation, so `b.Prepare()` clears the leak before the next spec; the spec that printed the warning, though, finished rendering in the emulated scheme. Read any odd-looking screenshot from that spec with that in mind. ### The `⚠` diagnostic notes **"Selector matched, then stopped matching"** — the detached node. The selector resolved, then its node was replaced (list re-key, portal migration) or its identifying attribute swapped in place. Silent when the selector genuinely never matched. ``` ⚠ Selector "#row-4" matched 6× during this poll (+0.00s to +0.41s) then stopped matching — the node was likely replaced, or its identifying attribute changed in place. ``` **"Click dispatched onto a covered element"** — fast `Click` stays occlusion-blind by design and still succeeds through an overlay, but records a hit-test so the downstream failure points somewhere. Diagnostic only; never changes whether a spec passes. ``` ⚠ Click on "#submit" was dispatched while <div#overlay.modal-scrim> was the topmost element at its centre — the click may have been swallowed. Consider Eventually("#submit").Should(b.BeClickable()) or b.Realistic().Click("#submit"). ``` **"Network handler never ran (shadowed by an earlier handler)"** — handlers are first-match-wins, so one registered for a URL an earlier handler claims is dead code. Deadly across an `Ordered` container, where `Prepare()` doesn't run between `It`s. Both call sites are named. ``` ⚠ A ModifyResponse handler registered at network_test.go:231 never ran — an earlier ModifyResponse handler (registered at network_test.go:223) claimed 1 matching response(s) first. ``` Reported only when a handler **never fired** *and* was shadowed at least once. **Limit:** it's a failure artifact, so it cannot surface shadowing's other presentation — a leftover *stateful* handler claiming the response, passing it through untouched, spec **green**. Only your own `Eventually(hold.Count).Should(Equal(1))` catches that (`flaky-specs` §6). **"Network handler could not read a response body"** — Chrome refuses to hand over some response bodies (a redirect's, notably), so a `ModifyResponse`/`HoldResponse` handler that matched never gets to run `Using`, apply a static override, or hold the response; Biloba continues it unmodified. Any other read failure (a timeout, a lost reply, a body over the limit) fails the request instead, since the body may already be gone. Either way the handler still counts the dispatch (`Count()` ticks at selection, before the read), and the note names the handler, its registration site, the URL, and which of the two Biloba did. ``` ⚠ A ModifyResponse handler registered at checkout_test.go:42 could not read the response body for http://localhost:8080/login (status 302): Can only get response body on requests captured after headers received. (-32000) Biloba continued the response unmodified. ``` ### When the failure is Chrome itself Every command Biloba sends Chrome runs under a deadline, so an unresponsive browser produces a failing spec instead of a hung suite. Three shapes, each naming its cause on the first line: | First line | Means | Do | |---|---|---| | `deadline_exceeded: Chrome did not <command> within 30s` | Chrome accepted the command and never answered | Chrome is wedged or the box is badly overloaded — look for a long synchronous script in the page, or too many parallel processes for the machine | | `page_crashed: this tab's renderer crashed` | Chrome reported the crash (promptly on macOS, several seconds later on Linux — a command in between reports `deadline_exceeded` instead) | usually the page itself (OOM, a bad WASM/canvas path). Navigating clears the crash and gets a fresh renderer on macOS; on Linux the tab has stayed dead in our testing, so continue on `b.NewTab()` rather than trusting the crashed one | | `frame_detached: the document this frame handle was found in is gone` | the frame handle's iframe was removed or navigated, or its tab navigated or `Prepare()`d, since `b.Frame` found it | a handle belongs to one document: call `b.Frame(...)` again after whatever replaces the frame, and don't carry a handle across that change | | `browser_gone: the connection to Chrome is closed` | the browser process exited — crashed, OOM-killed, or reaped | not recoverable; the rest of the suite will fail too. Check the machine's memory and whether anything is killing Chrome | The deadline is generous on purpose (a healthy command answers in milliseconds), so hitting it is a real signal, not a tight-timeout artifact. `WithTimeout` doesn't move it: that knob bounds how long Biloba keeps *retrying*, which is a different question from whether Chrome is alive. **A suite that ends on Ginkgo's `--timeout` with no failing spec** is a command blocked inside a poll's callback, where Gomega can't interrupt it. Biloba's own commands can't do that, so look for a `chromedp` call of your own on `b.Context` without a deadline. ### Two failure *messages* that self-explain - A two-axis getter timing out because the **property** never became defined says the element was present, names the property, and prints the `b.AllowMissing("disabled")` to paste (`flaky-specs` §5). - A failed `BePrecededBy`/`BeFollowedBy` reports the order actually observed (`Actually: #o-first comes BEFORE #o-second.`) — enough to spot an inverted assertion. ## Look at the page yourself, any time ```go fmt.Println(b.Outline()) // indented DOM fmt.Println(b.A11yOutline()) // accessibility tree: role + accessible name per node AddReportEntry("DOM before click", b.Outline(), ReportEntryVisibilityFailureOrVerbose) b.Run("document.querySelectorAll('.card').length") // quick count probe ``` `b.A11yOutline()` is **not** auto-attached — call it explicitly. It's often more useful than raw HTML for reasoning about what a page *means*. **Attach app/store state to every failure.** For a state-heavy or optimistic-UI app the store beats the DOM (which may be the pre-confirmation copy): ```go ReportAfterEach(func(report SpecReport) { if !report.Failed() { return } AddReportEntry("app state", b.Run(`JSON.stringify(window.__APP_STATE__ ?? null)`)) }) ``` Keep the `?? null` so a crashed page doesn't turn the snapshot itself into a failure. (That's a *snapshot*. To **wait** on app state as part of a spec, use `b.GetJSValue` — `flaky-specs` §3.) **Page `console.*` streams to the `GinkgoWriter`**, each argument rendered space-separated. Objects come from CDP's **shallow** preview, so nested/large objects log lossily. Build the string yourself when you need the whole value: `console.log('state ' + JSON.stringify(obj))`. ## Two headless quirks that look like Biloba bugs - **`HaveInnerText`/`GetInnerText` timing out on text that's plainly in the outline.** `innerText` is computed from layout and can return a stale/partial value before a paint settles. Switch to `HaveTextContent`/`GetTextContent` (reads `textContent` off the tree) or to a plain existence assertion. - **`requestAnimationFrame` never firing.** On a **fully static page**, `chrome-headless-shell` can leave rAF unscheduled after the first scroll (nothing animating ⇒ no frames), wedging app code driven off an rAF loop. Shape: a hang/timeout only in the default headless lane. Confirm with a counter read through a **coalescing `b.Run`** — `b.Run("window.__rafTicks ?? 0")`, **not** `b.GetJSValue`, which would sit waiting for a counter that never appears. Fix: drive the work off a real event, or run that spec under `HighFidelityHeadless()`/`BILOBA_INTERACTIVE=true`. ## Env vars | Var | Effect | |---|---| | `BILOBA_SCREENSHOTS_DIR=./artifacts` | where failure screenshots — and the visual `.actual.png`/`.diff.png` artifacts — are written | | `BILOBA_SCREENSHOT_BASELINES_DIR=./baselines` | where `b.HaveScreenshot`'s **committed** baselines live (default `./biloba-baselines`) | | `BILOBA_UPDATE_SCREENSHOTS=1` | capture and (re)write every visual baseline the run touches instead of comparing; prints what changed. Accepts `1/t/true/y/yes/on` (off: `0/f/false/n/no/off`), case-insensitive; **any other value warns and is treated as off**. Suite-wide — scope it with `--focus`. **Never set it in CI**: every visual assertion then passes unconditionally | | `BILOBA_OUTLINE_MAX=131072` | raise the outline byte cap; `0`/`off` = no truncation | | `BILOBA_INLINE_SCREENSHOTS=iterm\|kitty\|sixel\|none` | force an inline-image protocol, or `none` to disable the blob (the file path is still printed — use `none` in CI and in Claude Code, where base64 is noise) | | `BILOBA_PROBE_TERMINAL=true` | actively query the TTY for Sixel support when env detection finds nothing | | `BILOBA_INTERACTIVE=true` | headful high-fidelity run that pauses on failure until `^C` | Inline images are auto-detected (Kitty, iTerm2, Sixel/VS Code) and only emitted when the terminal supports them. ## Config knobs (`ConnectToChrome`) Each boolean takes an optional bool (no arg = `true`). **Explicit settings win, per knob** — automation only fills knobs you left untouched. - `BilobaConfigScreenshotsToDir(dir)` — write each tab's failure screenshot there (prints the absolute path). - `BilobaConfigFailureOutlines(...bool)` / `BilobaConfigInlineScreenshots(...bool)` — force on/off. - `BilobaConfigFailureScreenshots(...bool)` (default on) / `BilobaConfigPollTrajectory(...bool)` (default on) / `BilobaConfigProgressReportScreenshots(...bool)` (default on). - `BilobaConfigFailureScreenshotsSize(w,h)` / `BilobaConfigProgressReportScreenshotSize(w,h)`. - `BilobaConfigDebugLogging(...bool)` — stream all CDP traffic to the `GinkgoWriter` (verbose). - `BilobaConfigScreenshotBaselinesDir(dir)` — where `b.HaveScreenshot` reads/writes committed baselines (default `./biloba-baselines`; commit that dir, gitignore the screenshots dir). - `BilobaConfigScreenshotTolerance(fraction)` / `BilobaConfigScreenshotChannelTolerance(delta)` — suite-wide visual-comparison defaults, both `0` (exact) by default; `b.Tolerance(...)`/`b.ChannelTolerance(...)` override per assertion. → `visual-assertions` ```go // CI that only redirects the directory still keeps the automation default of outlines-on: b = biloba.ConnectToChrome(GinkgoT(), biloba.BilobaConfigScreenshotsToDir("./artifacts")) ``` ## Watch it live ```bash BILOBA_INTERACTIVE=true ginkgo --focus="..." ``` Headful, high fidelity, prints the failure and waits for `^C`. Use a small handful of focused specs, in serial. (`SpinUpChrome(GinkgoT(), biloba.ChromeFlags(chromedp.Flag("headless", false)))` does the same in code.) ## A hang, not a failure Biloba screenshots Ginkgo [progress reports](https://onsi.github.io/ginkgo/#getting-visibility-into-long-running-specs) — on a spec timeout, a `PollProgressAfter` spec, or on demand: `^T` (SIGINFO) on macOS, `SIGUSR2` on Linux.