TestBox Browser Testing
When to Use This Skill
- Writing specs that drive a real browser (Chromium, Firefox or WebKit) against a running web app
- Choosing between
BrowserSpec, ColdBox'sBrowserTestCaseand plainBaseSpec - Asserting on pages and locators with
expect( page ).toSee( "Welcome" )style matchers - Getting screenshots, traces and videos of failed specs into the test report
- Retrying flaky specs, rerunning only failures, or letting the runner start the web server
- Setting up browser tests in CI and debugging a failed run
Requirements and Engine Support
testbox.system.BrowserSpec is built on the bx-playwright module, so it is BoxLang only (BoxLang 1.17+, Java 21+, TestBox 7.2.0+ with testbox.system.browser).
- When the engine is not BoxLang, or bx-playwright is not installed,
browse()andthis.playwright()skip the running spec with the reason (bx-playwright is not installed: install-bx-module bx-playwright), so the rest of the suite still runs. - Keep browser specs as
.bxfiles in their own folder (for exampletests/specs/browser) and exclude that folder on Lucee or Adobe, which cannot compile BoxLang classes. - Attachments, retries,
--failedandPlaywright.AssertionFailedcounted as a failure work for every spec on every engine.
| You test... | Extend |
|---|---|
| Any web app from TestBox | testbox.system.BrowserSpec |
A ColdBox app (named routes, loginAs()) |
coldbox.system.testing.BrowserTestCase, see the coldbox-testing-browser skill |
| Any other BoxLang spec | testbox.system.BaseSpec plus testbox.system.browser.BrowserSupport (see below) |
For CFML engines, the separate cbplaywright module (see modules/cbplaywright) is the alternative.
Install bx-playwright and a Browser
# The module and its bxPlaywright CLI in the BoxLang OS runtime
install-bx-module bx-playwright
# Playwright driver, Node.js and Chromium, then check the setup
bxPlaywright install chromium
bxPlaywright doctor
# Linux CI machines: also install the system libraries the browser needs
bxPlaywright install chromium --with-deps
Everything downloads to ~/.boxlang/playwright. The module must be loaded by the runtime that runs the tests: the BoxLang OS runtime for ./testbox/run, or the CommandBox BoxLang server for the HTML runner (for example "onServerInitialInstall": "install bx-playwright --noSave" in server.json).
Your First Browser Spec
// tests/specs/browser/LoginSpec.bx
class extends="testbox.system.BrowserSpec" baseURL="http://localhost:8080" browserProfile="ci" {
function run() {
describe( "Login", () => {
it( "signs in with valid credentials", () => {
browse( ( page ) => {
page.visit( "/login" )
.fill( "Email", "luis@ortus.com" )
.fill( "Password", "secret" )
.click( "Sign in" )
expect( page ).toHavePath( "/dashboard" )
expect( page ).toSee( "Welcome back" )
} )
} )
it( "rejects a wrong password", () => {
browse( ( page ) => {
page.visit( "/login" )
.fill( "Email", "luis@ortus.com" )
.fill( "Password", "nope" )
.click( "Sign in" )
expect( page ).toHavePath( "/login" )
expect( page.locator( "@error" ) ).toHaveText( "Invalid credentials" )
} )
} )
} )
}
}
xUnit style works the same: function testSignsIn() { browse( ( page ) => { ... } ) }.
The page API (visit(), fill(), click(), locators, finders, assertSee()...) is bx-playwright's. Both assertion styles mix freely: TestBox browser matchers (expect( page ).toSee()) and bx-playwright inline assertions (page.assertSee(), page.assertPathIs()). A Playwright.AssertionFailed counts as a spec failure; other Playwright.* exceptions count as errors.
How a BrowserSpec Works
- One browser per bundle, started on first use and closed after the bundle by
closeBrowser(), which carries theafterAllannotation. Your ownbeforeAll()/afterAll()need nosupercalls. - Fresh pages per
browse()call, each in its own browser context (own cookies, storage, session), closed when the callback ends. - Browser matchers registered for every spec of the bundle.
- Failure artifacts attached: when the callback throws, the kept screenshots, trace and videos are attached to the spec and the exception is rethrown unchanged.
- Not thread safe: never use
asyncAllin suites that browse.
Class Annotations
| Annotation | Description |
|---|---|
baseURL |
URL that relative visits such as page.visit( "/login" ) resolve against |
browserProfile |
bx-playwright profiles for the bundle browser, a list such as ci or ci,mobile (merged left to right) |
retries |
Extra attempts for failing specs of the bundle (see Retries) |
baseURL and browserProfile are also found on parent classes, so a shared base spec can set them once.
baseURL resolution order:
- The
baseURLannotation of the spec class (or a class it extends) - The
--web-server-urlof the BoxLang runner when it started a web server (kept inserver.testbox.webServerURL) - The bx-playwright
baseURLsetting or theBX_PLAYWRIGHT_BASEURLenvironment variable
Without browserProfile, bx-playwright uses its defaultProfile setting or BX_PLAYWRIGHT_PROFILE, so BX_PLAYWRIGHT_PROFILE=ci is a handy CI switch.
browse( callback, options = {} )
Runs the callback with fresh pages: one page per declared callback argument, each in its own browser context. A callback with no declared arguments receives one page as its first positional argument. It returns the callback result.
// One page
browse( ( page ) => page.visit( "/" ).assertSee( "Welcome" ) )
// Return a value out of the page
var title = browse( ( page ) => page.visit( "/" ).title() )
expect( title ).toInclude( "Home" )
// Two users signed in at once, no shared cookies
browse( ( alice, bob ) => {
alice.visit( "/login" ).fill( "Email", "alice@example.com" ).fill( "Password", "secret" ).click( "Sign in" )
bob.visit( "/login" ).fill( "Email", "bob@example.com" ).fill( "Password", "secret" ).click( "Sign in" )
alice.visit( "/chat" ).fill( "Message", "Hi Bob" ).click( "Send" )
expect( bob.visit( "/chat" ) ).toSee( "Hi Bob" )
} )
// Second argument: bx-playwright newContext() options for every page of the call
browse(
( page ) => {
page.visit( "/" )
expect( page.locator( "@mobile-menu" ) ).toBeVisible()
},
{ viewport : { width : 390, height : 844 }, timeouts : { assertion : 10000 } }
)
this.playwright() and browserAvailable()
// The bundle manager, created with the bundle annotations and closed after the bundle
var response = this.playwright().request().get( "/api/products" )
expect( response.status() ).toBe( 200 )
var ctx = this.playwright().newContext() // manual contexts when browse() is not enough
Always call it as
this.playwright(). An unqualifiedplaywright()resolves to the bx-playwright BIF, even inside aBrowserSpec, and returns a new manager that ignores the bundle annotations and is not closed for you.
browserAvailable() is true on BoxLang with bx-playwright loaded. Use it to skip whole suites up front:
describe( title = "Checkout", skip = !browserAvailable(), body = () => {
it( "pays with a card", () => {
browse( ( page ) => { ... } )
} )
} )
Browser Matchers
Registered automatically for BrowserSpec and ColdBox BrowserTestCase bundles. In any other BoxLang spec:
addMatchers( new testbox.system.browser.BrowserMatchers() )
| Matcher | Target | Passes when |
|---|---|---|
toHaveTitle( title ) |
page | Title is exactly title, or matches a page.regex() |
toHaveURL( url ) |
page | Full URL is exactly url, or matches a page.regex() |
toHavePath( path ) |
page | URL path equals path (case sensitive), ignoring scheme, host, query and hash |
toSee( text ) |
page or locator | Text (or regex) is visible, like assertSee() |
toHaveText( text ) |
locator | Text of the first element is exactly text (whitespace normalized), or matches a regex |
toBeVisible() |
locator | First element is visible |
toBeHidden() |
locator | First element is hidden or not in the page |
toHaveCount( count ) |
locator | Locator matches exactly count elements |
toHaveValue( value ) |
locator | Input, textarea or select value equals value, or matches a regex |
Every matcher has a not form: prefix with not (notToSee, notToHaveTitle, notToBeVisible, notToHaveCount...).
expect( page ).toHaveTitle( "Dashboard" )
expect( page ).toHaveTitle( page.regex( "^Dash" ) )
expect( page ).toHaveURL( "http://localhost:8080/dashboard?tab=1" )
expect( page ).toHavePath( "/dashboard" )
expect( page ).toSee( "Welcome" )
expect( page ).notToSee( "Error" )
expect( page.locator( "nav" ) ).toSee( "Logout" )
expect( page.locator( "h1" ) ).toHaveText( "Todos" )
expect( page.locator( ".todo" ) ).toHaveCount( 3 )
expect( page.locator( "@spinner" ) ).toBeHidden()
expect( page.locator( "@error" ) ).notToBeVisible()
expect( page.locator( "#email" ) ).toHaveValue( "luis@ortus.com" )
- Web-first: they delegate to bx-playwright's retrying assertions and wait until they pass or the assertion timeout (
timeouts.assertion, 5000 ms by default) expires. Negated forms wait too. Neversleep()before them. - A failure is a normal TestBox failure carrying bx-playwright's message (expected, received, call log).
- They refuse values that are not bx-playwright pages or locators, except
toHavePath(), which falls back to the core Data NavigatortoHavePath()for structs and other data.
Attachments
Automatic browser artifacts
When a browse() callback throws, its contexts close with failed = true, so bx-playwright's artifact policies keep their files, and the kept files are attached to the spec with types screenshot, trace and video. Turn artifacts on with a profile (browserProfile="ci", record or debug) or per call:
browse( ( page ) => { ... }, {
artifacts : { screenshot : "only-on-failure", trace : "retain-on-failure", video : "retain-on-failure" }
} )
Policies: off, on, only-on-failure, retain-on-failure.
attach( path, type = "file", name = "" )
Any spec (browser or not, any engine) can attach files to the running spec. name defaults to the file name of path.
it( "exports the report", () => {
var file = service.export()
attach( file, "file", "export.csv" )
attach( logPath, "log" )
expect( fileExists( file ) ).toBeTrue()
} )
- Call it from a spec body or a
beforeEach()/afterEach()/aroundEach()closure. Outside a running spec it throwsTestBox.InvalidContext. - Attachments are kept for passed and failed specs in the
attachmentsarray of the spec stats. - Reporters: the JSON report includes them, the Simple report links them, JUnit and ANTJunit add a
<system-out>with one[[ATTACHMENT|path]]line per file, and text, console and stream outputs list them under failed specs.
Retries
A failing or erroring spec reruns up to N extra times; only the final attempt is recorded. BDD reruns every beforeEach(), the aroundEach() closures, the body and afterEach(); xUnit reruns setup(), the test and teardown(). Skipped specs are never retried. Output shows "(passed after N attempts)" and the attempts spec stat counts the runs. Attachments of earlier attempts stay on the spec.
// 1. Spec argument (wins), also on fit() and xit()
it( title = "talks to a flaky service", retries = 2, body = () => { ... } )
// 2. Bundle annotation
class extends="testbox.system.BrowserSpec" retries="1" { ... }
// xUnit: method annotation, wins over the bundle annotation
function testCheckout() retries="2" { ... }
# 3. Global default (lowest precedence)
./testbox/run --retries=2
Programmatically: new testbox.system.TestBox( directory = "tests.specs", options = { retries : 2 } ).
Retries hide flakiness, they do not fix it: prefer the web-first matchers, which already wait, and keep the count low.
Running Browser Tests
# Run the browser folder
./testbox/run --directory=tests.specs.browser
# Rerun only what failed or errored in the last run
./testbox/run --failed
# Let the runner start the app, wait for it, run the tests and stop it
./testbox/run --directory=tests.specs.browser \
--web-server="boxlang-miniserver --port 8080" \
--web-server-url=http://localhost:8080 \
--web-server-timeout=60
| Option | Default | Description |
|---|---|---|
--retries |
0 |
Extra runs for failing or erroring specs (spec and bundle values win) |
--failed |
false |
Run only bundles and specs listed in {reportpath}/.testbox-failed.json, which every run writes. When missing or empty, prints a message and runs nothing |
--web-server |
Shell command started before the tests (sh -c, cmd /c on Windows), stopped with its child processes after them |
|
--web-server-url |
http://localhost:8080 |
Polled until it answers with a status below 500; becomes the default baseURL of BrowserSpec bundles |
--web-server-timeout |
60 |
Seconds to wait; when the server does not answer the runner stops it and exits with code 1 |
If the server command has its own flags that share a runner option name (for example --directory), put it in a script.
CI (GitHub Actions)
jobs:
browser-tests:
runs-on: ubuntu-latest
env:
BX_PLAYWRIGHT_PROFILE: ci # headless, keep artifacts of failures
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with: { distribution: temurin, java-version: "21" }
- uses: ortus-boxlang/setup-boxlang@main
with:
version: latest
modules: bx-playwright
- uses: actions/cache@v4
with:
path: ~/.boxlang/playwright
key: playwright-${{ runner.os }}
- name: Install Chromium
run: |
export PATH="$HOME/.boxlang/bin:$PATH"
bxPlaywright install chromium --with-deps
- name: Run the browser tests
run: |
./testbox/run --directory=tests.specs.browser \
--web-server="boxlang-miniserver --port 8080" \
--web-server-url=http://127.0.0.1:8080 \
--retries=1 --reporter=junit
- name: Upload failure artifacts
if: failure()
uses: actions/upload-artifact@v4
with:
name: browser-test-artifacts
path: |
~/.boxlang/playwright/artifacts
tests/results
if-no-files-found: ignore
Run browser specs in their own job so they do not slow down unit and integration tests, and cache ~/.boxlang/playwright to skip the browser download.
Debugging
- Open a trace from a failed spec (or a CI artifact):
bxPlaywright show-trace path/to/trace.zip. - Watch the browser:
browserProfile="debug"(headed, slowMo, all artifacts) orBX_PLAYWRIGHT_HEADLESS=false. - Inspect the page structure with
page.snapshot(); record selectors withbxPlaywright codegen <url>. - A spec reported as skipped with an install hint means bx-playwright is not loaded in the runtime running the tests: run
bxPlaywright doctor.
Using the Browser From Any BoxLang Spec
BrowserSpec delegates to testbox.system.browser.BrowserSupport; use it directly when you cannot change the base class, and close it yourself:
class extends="testbox.system.BaseSpec" {
function beforeAll() {
variables.browser = new testbox.system.browser.BrowserSupport( this )
addMatchers( new testbox.system.browser.BrowserMatchers() )
}
function afterAll() {
variables.browser.close()
}
function run() {
describe( "Home page", () => {
it( "greets visitors", () => {
variables.browser.browse( ( page ) => {
page.visit( "http://localhost:8080/" )
expect( page ).toSee( "Welcome" )
} )
} )
} )
}
}
Related Skills
coldbox-testing-browser:BrowserTestCasefor ColdBox apps (named routes,loginAs())testbox-runners,testbox-reporters,testbox-expectations,testbox-bdd,testbox-unit- bx-playwright skills (
bx-playwright-browsing,bx-playwright-assertions,bx-playwright-testing) for the full page API, page objects and visual regression