Debug Failures

Debugging Biloba failures

Reading artifacts after a spec failed. To prevent flakes → flaky-specs. Docs: https://onsi.github.io/biloba/#failure-artifacts-humans-ci-and-agents. Sibling skills are named here without a prefix; invoke one with the same plugin prefix you loaded this skill under.

Zero config: what you already get

Biloba detects the environment in ConnectToChrome. "Automation" = CI is set or an AI coding agent is detected (CLAUDECODE/AI_AGENT/Cursor/Gemini CLI/Codex/…).

So an agent or CI run needs nothing: ginkgo -r -p, then read the outline and the screenshot files. BILOBA_SCREENSHOTS_DIR=./artifacts points the directory elsewhere.

Read the artifacts in this order

1. Console errors — any console.error/console.assert before the failure, replayed under "Console errors logged before this failure" at the top of the failure block. On a JS crash (a React error boundary) this is the root cause.
2. ⚠ diagnostic notes — they name the cause outright.
3. Poll trajectory — what the read you failed on did over its whole deadline (absent when the failure had no value read behind it).
4. Visual diagnosis — only on a failed b.HaveScreenshot: pixel counts plus the shape of the change, in words, before you open any image.
5. Screenshot — Read the printed PNG path.
6. DOM outline — "DOM Outline for: ''": indented DOM, <script>/<style>/<svg> bodies pruned, whitespace collapsed, capped at 32 KB. Past ... [truncated]? Raise the cap with BILOBA_OUTLINE_MAX — a byte count (=131072), or 0/off for the whole DOM.

Poll trajectory

When an Eventually over a read that observes and compares a value (b.HaveInnerText/b.HaveCount/any value matcher, a geometry matcher, b.EvaluateTo, b.GetJSValue) times out, Biloba attaches the (elapsed, value) series of that assertion. Gomega's Timed out … Expected <120> shows only the final value; the shape is the diagnosis:

No entry is a normal outcome, not a bug. The series is claimed by the matcher Gomega asked for a failure message, so an entry always describes the read you failed on. Reads that passed, b.Run setup lines, and failures with no value read underneath (a b.Click whose selector never matched, a getter whose value was never there — that one gets the AllowMissing enrichment instead) simply produce nothing. Don't read a missing trajectory as a signal; go to the outline and the screenshot. To get a trajectory for an arbitrary expression, poll it with b.EvaluateTo/b.GetJSValue rather than wrapping b.Run in your own Eventually — Biloba only records reads it owns.

On by default; BilobaConfigPollTrajectory(false) disables it (and the detached-node signal). Don't disable it for speed. Recording takes a lock and renders the value per poll sample, so it isn't free — but measured on a 1,558-spec suite with 2,470 call sites across the instrumented matchers (including gates asserting on whole paragraphs of prose), turning it off saved under a second across the whole run, inside that suite's own run-to-run spread.

Visual diagnosis (a failed b.HaveScreenshot) → visual-assertions

Two extra PNGs land in the screenshots dir — <name>.actual.png (what Biloba saw) and <name>.diff.png (the actual, washed out, differing pixels in magenta) — and the failure message says what moved:

screenshot "home-desktop" differs from baseline
  38,160 of 1,017,600 pixels differ (3.75%), max channel delta 221
  changed region: one box, (0,14)-(1272,44)  [100% of the width, 4% of the height, at its top edge]
  unchanged: everything below y=44
  baseline: /Users/you/app/biloba-baselines/home-desktop.png
  actual:   /Users/you/app/biloba-screenshots/home-desktop.actual.png
  diff:     /Users/you/app/biloba-screenshots/home-desktop.diff.png

The shape line (one box, changed regions: N boxes, scattered, uniform shift, a size change) and what to do about each → visual-assertions. When every differing pixel is off by only a few levels Biloba says a rasterisation or compositing difference, not a content change — nothing moved, so look for a shadow or gradient compositing into the capture. A missing baseline is a different failure: it says to re-run with BILOBA_UPDATE_SCREENSHOTS=1; never script your way past it. Read the .diff.png when the words aren't enough (inline images are off under an agent, so you get the path).

"never settled" is printed during an update run, not on a failure. The run stays green, but the baseline it just wrote was captured from a page still changing and the next normal run will fail against it. Mask the moving region (or stop it moving) and re-run the update; re-running alone changes nothing. → visual-assertions

"Failed to clear the emulated prefers-color-scheme" — a dropped b.InColorSchemes teardown. The override is target-level and survives navigation, so b.Prepare() clears the leak before the next spec; the spec that printed the warning, though, finished rendering in the emulated scheme. Read any odd-looking screenshot from that spec with that in mind.

The ⚠ diagnostic notes

"Selector matched, then stopped matching" — the detached node. The selector resolved, then its node was replaced (list re-key, portal migration) or its identifying attribute swapped in place. Silent when the selector genuinely never matched.

⚠ Selector "#row-4" matched 6× during this poll (+0.00s to +0.41s) then stopped matching
  — the node was likely replaced, or its identifying attribute changed in place.

"Click dispatched onto a covered element" — fast Click stays occlusion-blind by design and still succeeds through an overlay, but records a hit-test so the downstream failure points somewhere. Diagnostic only; never changes whether a spec passes.

⚠ Click on "#submit" was dispatched while <div#overlay.modal-scrim> was the topmost
  element at its centre — the click may have been swallowed.
  Consider Eventually("#submit").Should(b.BeClickable()) or b.Realistic().Click("#submit").

"Network handler never ran (shadowed by an earlier handler)" — handlers are first-match-wins, so one registered for a URL an earlier handler claims is dead code. Deadly across an Ordered container, where Prepare() doesn't run between Its. Both call sites are named.

⚠ A ModifyResponse handler registered at network_test.go:231 never ran — an earlier ModifyResponse
  handler (registered at network_test.go:223) claimed 1 matching response(s) first.

Reported only when a handler never fired and was shadowed at least once. Limit: it's a failure artifact, so it cannot surface shadowing's other presentation — a leftover stateful handler claiming the response, passing it through untouched, spec green. Only your own Eventually(hold.Count).Should(Equal(1)) catches that (flaky-specs §6).

When the failure is Chrome itself

Every command Biloba sends Chrome runs under a deadline, so an unresponsive browser produces a failing spec instead of a hung suite. Three shapes, each naming its cause on the first line:

The deadline is generous on purpose (a healthy command answers in milliseconds), so hitting it is a real signal, not a tight-timeout artifact. WithTimeout doesn't move it: that knob bounds how long Biloba keeps retrying, which is a different question from whether Chrome is alive.

A suite that ends on Ginkgo's --timeout with no failing spec is a command blocked inside a poll's callback, where Gomega can't interrupt it. Biloba's own commands can't do that, so look for a chromedp call of your own on b.Context without a deadline.

Two failure messages that self-explain

• A two-axis getter timing out because the property never became defined says the element was present, names the property, and prints the b.AllowMissing("disabled") to paste (flaky-specs §5).
• A failed BePrecededBy/BeFollowedBy reports the order actually observed (Actually: #o-first comes BEFORE #o-second.) — enough to spot an inverted assertion.

Look at the page yourself, any time

fmt.Println(b.Outline())     // indented DOM
fmt.Println(b.A11yOutline()) // accessibility tree: role + accessible name per node
AddReportEntry("DOM before click", b.Outline(), ReportEntryVisibilityFailureOrVerbose)
b.Run("document.querySelectorAll('.card').length")   // quick count probe

b.A11yOutline() is not auto-attached — call it explicitly. It's often more useful than raw HTML for reasoning about what a page means.

Attach app/store state to every failure. For a state-heavy or optimistic-UI app the store beats the DOM (which may be the pre-confirmation copy):

ReportAfterEach(func(report SpecReport) {
    if !report.Failed() { return }
    AddReportEntry("app state", b.Run(`JSON.stringify(window.__APP_STATE__ ?? null)`))
})

Keep the ?? null so a crashed page doesn't turn the snapshot itself into a failure. (That's a snapshot. To wait on app state as part of a spec, use b.GetJSValue — flaky-specs §3.)

Page console.* streams to the GinkgoWriter, each argument rendered space-separated. Objects come from CDP's shallow preview, so nested/large objects log lossily. Build the string yourself when you need the whole value: console.log('state ' + JSON.stringify(obj)).

Two headless quirks that look like Biloba bugs

• HaveInnerText/GetInnerText timing out on text that's plainly in the outline. innerText is computed from layout and can return a stale/partial value before a paint settles. Switch to HaveTextContent/GetTextContent (reads textContent off the tree) or to a plain existence assertion.
• requestAnimationFrame never firing. On a fully static page, chrome-headless-shell can leave rAF unscheduled after the first scroll (nothing animating ⇒ no frames), wedging app code driven off an rAF loop. Shape: a hang/timeout only in the default headless lane. Confirm with a counter read through a coalescing b.Run — b.Run("window.__rafTicks ?? 0"), not b.GetJSValue, which would sit waiting for a counter that never appears. Fix: drive the work off a real event, or run that spec under HighFidelityHeadless()/BILOBA_INTERACTIVE=true.

Env vars

Inline images are auto-detected (Kitty, iTerm2, Sixel/VS Code) and only emitted when the terminal supports them.

Config knobs (ConnectToChrome)

Each boolean takes an optional bool (no arg = true). Explicit settings win, per knob — automation only fills knobs you left untouched.

• BilobaConfigScreenshotsToDir(dir) — write each tab's failure screenshot there (prints the absolute path).
• BilobaConfigFailureOutlines(...bool) / BilobaConfigInlineScreenshots(...bool) — force on/off.
• BilobaConfigFailureScreenshots(...bool) (default on) / BilobaConfigPollTrajectory(...bool) (default on) / BilobaConfigProgressReportScreenshots(...bool) (default on).
• BilobaConfigFailureScreenshotsSize(w,h) / BilobaConfigProgressReportScreenshotSize(w,h).
• BilobaConfigDebugLogging(...bool) — stream all CDP traffic to the GinkgoWriter (verbose).
• BilobaConfigScreenshotBaselinesDir(dir) — where b.HaveScreenshot reads/writes committed baselines (default ./biloba-baselines; commit that dir, gitignore the screenshots dir).
• BilobaConfigScreenshotTolerance(fraction) / BilobaConfigScreenshotChannelTolerance(delta) — suite-wide visual-comparison defaults, both 0 (exact) by default; b.Tolerance(...)/b.ChannelTolerance(...) override per assertion. → visual-assertions

// CI that only redirects the directory still keeps the automation default of outlines-on:
b = biloba.ConnectToChrome(GinkgoT(), biloba.BilobaConfigScreenshotsToDir("./artifacts"))

Watch it live

BILOBA_INTERACTIVE=true ginkgo --focus="..."

Headful, high fidelity, prints the failure and waits for ^C. Use a small handful of focused specs, in serial. (SpinUpChrome(GinkgoT(), biloba.ChromeFlags(chromedp.Flag("headless", false))) does the same in code.)

A hang, not a failure

Biloba screenshots Ginkgo progress reports — on a spec timeout, a PollProgressAfter spec, or on demand: ^T (SIGINFO) on macOS, SIGUSR2 on Linux.