bx-playwright: Testing, Page Objects and Quality
BoxLang is the AI-native software productivity platform for building, modernizing and running applications, with developers and AI agents working together.
Browser support is turned on by a class annotation, not a base class. Any spec whose class (or a parent class) has @browser, @browserProfile( ... ) or @baseURL( ... ) gets it:
| You test... | Write | Gives you |
|---|---|---|
| Any web app from TestBox | @browser + class extends="testbox.system.BaseSpec" |
browse(), bundle browser, browser matchers, attachments |
| A ColdBox app | @browser + class extends="coldbox.system.testing.BaseTestCase" |
All of the above plus routeURL(), visitRoute(), assertRouteIs() |
| Older TestBox, or another framework | testbox.system.BaseSpec |
playwright( "ci" ).browse() by hand (see Plain TestBox) |
testbox.system.BrowserSpec still exists as an optional base class: it is just @browser on BaseSpec. ColdBox's BrowserTestCase was removed: use BaseTestCase with @browser.
All of them are BoxLang only. When bx-playwright is not installed (or the engine is not BoxLang), browser specs are skipped with an install hint, not failed.
TestBox @browser Specs (recommended)
// tests/specs/browser/LoginSpec.bx
@browser
@baseURL( "http://localhost:8080" )
@browserProfile( "ci" )
class extends="testbox.system.BaseSpec" {
function run() {
describe( "Login", () => {
it( "signs in", () => {
browse( ( page ) => {
page.visit( "/login" )
.fill( "Email", "luis@ortus.com" )
.fill( "Password", "secret" )
.click( "Sign in" )
expect( page ).toHavePath( "/dashboard" )
expect( page ).toSee( "Welcome" )
} )
} )
} )
}
}
Class annotations, written as BoxLang annotations above class (never inline attributes in .bx files). Any of the first three turns browser support on:
browser: browser support with the bx-playwright default profile.browserProfile: bx-playwright profiles for the bundle browser, a list such asciorci,mobile.baseURL: relativevisit()calls resolve against it. Default: the runner's--web-server-urlwhen the runner started a web server, else bx-playwright's ownbaseURLsetting orBX_PLAYWRIGHT_BASEURL.browserAutoInstall( false ): do not download a missing browser on first use (pre-provisioned or offline CI); callensureBrowserInstalled()yourself if needed.
The runner mixes these into the spec (this and variables scopes, so call them unqualified): browse(), browserAvailable(), ensureBrowserInstalled(), getBrowserSupport(), closeBrowser(), plus this.playwright(), and registers the browser matchers. Methods your spec declares itself are kept.
How it behaves:
- One browser per bundle, started on first use. The runner closes it after the bundle, even when
afterAll()throws, so your ownbeforeAll()/afterAll()need nosupercalls for it. - Safety net: when the request aborts (a spec calls
abort), the browser stays open until the next test run that opens a browser, which closes it first. bx-playwright closes any instance still open when the module unloads or the JVM shuts down. - Every
browse()call gets fresh pages, each in its own isolated browser context, closed when the callback ends. browse()returns the callback result.- Do not use
asyncAllin suites that browse: browser specs are not thread safe.
browse()
browse( ( page ) => page.visit( "/" ).assertSee( "Welcome" ) )
// One isolated page (own cookies and storage) per declared argument
browse( ( admin, guest ) => {
admin.visit( "/admin" )
guest.visit( "/admin" )
expect( guest ).notToSee( "Dashboard" )
} )
// Second argument: newContext() options for this call
browse( ( page ) => {
page.visit( "/" )
}, { viewport : { width : 390, height : 844 }, timeouts : { assertion : 2000 } } )
this.playwright(), browserAvailable() and getBrowserSupport()
pw = this.playwright() // the bundle manager (closed for you after the bundle)
ctx = pw.newContext() // manual contexts when browse() is not enough
// playwright() without `this.` is the bx-playwright BIF: a NEW manager the bundle does not close
it( title = "needs a browser", skip = !browserAvailable(), body = () => { ... } )
getBrowserSupport().getUnavailableReason() // why browser specs cannot run here, "" when they can
Browser Matchers
Registered automatically for browser-enabled bundles (@browser, @browserProfile, @baseURL). In any other BoxLang spec: addMatchers( new testbox.system.browser.BrowserMatchers() ).
| Matcher | Target | Checks |
|---|---|---|
toHaveTitle( title ) |
page | Exact title, or a page.regex() |
toHaveURL( url ) |
page | Full URL, or a page.regex() |
toHavePath( path ) |
page | URL path only (ignores scheme, host, query, hash), case sensitive |
toSee( text ) |
page or locator | Visible text (like assertSee()) |
toHaveText( text ) |
locator | Exact text of the first element (whitespace normalized), or regex |
toBeVisible() |
locator | First element visible |
toBeHidden() |
locator | First element hidden or absent |
toHaveCount( count ) |
locator | Exact number of matches |
toHaveValue( value ) |
locator | Input, textarea or select value, or regex |
Every matcher has a not form (notToSee, notToHaveCount, notToBeVisible, ...).
expect( page ).toHaveTitle( "Dashboard" )
expect( page ).toHaveTitle( page.regex( "^Dash" ) )
expect( page ).toHaveURL( "http://localhost:8080/dashboard?tab=1" )
expect( page ).toHavePath( "/dashboard" )
expect( page ).toSee( "Welcome" )
expect( page ).notToSee( "Error" )
expect( page.locator( "h1" ) ).toHaveText( "Todos" )
expect( page.locator( ".todo" ) ).toHaveCount( 3 )
expect( page.locator( "@spinner" ) ).toBeHidden()
expect( page.locator( "@error" ) ).notToBeVisible()
expect( page.locator( "#email" ) ).toHaveValue( "luis@ortus.com" )
- They are web-first: they retry until they pass or the bx-playwright assertion timeout (
timeouts.assertion, 5000 ms) expires. Negated forms wait too. Neversleep()before them. - A failure is a TestBox failure that carries bx-playwright's message (expected, received, call log).
- A matcher refuses values that are not bx-playwright pages or locators, except
toHavePath(), which falls back to TestBox's Data NavigatortoHavePath()for structs and other data. - bx-playwright's own inline assertions (
page.assertSee(),page.expect().toHaveText()) also work in specs:Playwright.AssertionFailedcounts as a spec failure (otherPlaywright.*errors count as errors).
Automatic Attachments
When a browse() callback throws, its contexts close with failed = true, so the artifact policies keep their files, and the kept screenshots, trace and videos are attached to the spec with attach() (types screenshot, trace, video). The exception is rethrown unchanged. Reporters list attachments: JSON report, links in the Simple report, [[ATTACHMENT|path]] lines in JUnit <system-out>, and text/console/stream output under failed specs.
Turn artifacts on with a profile (@browserProfile( "ci" )) or per call:
browse( ( page ) => { ... }, {
artifacts : { screenshot : "only-on-failure", trace : "retain-on-failure", video : "retain-on-failure" }
} )
Attach your own files from any spec: attach( path, type = "file", name = "" ), for example attach( logPath, "log" ).
Retries, --failed and --web-server
it( title = "flaky checkout", retries = 2, body = () => { ... } ) // spec wins
@browser
@retries( 1 ) // bundle annotation, above class
class extends="testbox.system.BaseSpec" { ... }
./testbox/run --directory=tests.specs.browser --retries=2 # global default
./testbox/run --failed # rerun only what failed last run
./testbox/run --web-server="boxlang-miniserver --port 8080" \
--web-server-url=http://localhost:8080 --web-server-timeout=60
- Precedence:
it( ..., retries )> bundleretriesannotation >--retries. A retry rerunsbeforeEach(), body andafterEach(). Skipped specs are never retried. Output shows "(passed after N attempts)". --failedreads{reportpath}/.testbox-failed.json, which every run writes. Bundles that failed outside of a spec (beforeAll(),afterAll()) are listed underbundleErrorsand never rerun: fix them and run them directly. In the web runner, the HTML reports offer a Run Failed (N) button instead, built from the report (no state). Both useTestResult.getFailedTargets().--web-serverstarts the command, waits until--web-server-urlanswers (status below 500), runs the tests, then stops the server and its child processes. It exits with code 1 when the server does not answer in time. The URL becomes the defaultbaseURLof browser-enabled bundles.
ColdBox: BaseTestCase with @browser
coldbox.system.testing.BaseTestCase loads your ColdBox app like any integration test (appMapping, webMapping, configMapping, ...); add @browser to drive a real browser against the running app. Everything in the TestBox section above applies. (BrowserTestCase was removed; browserUnavailableReason() is now getBrowserSupport().getUnavailableReason().)
@appMapping( "/root" )
@browser
@baseURL( "http://127.0.0.1:8080" )
@browserProfile( "ci" )
class extends="coldbox.system.testing.BaseTestCase" {
function beforeAll() {
super.beforeAll()
// Log in once through the real login page, reused by browse( ..., { session : "admin" } )
this.playwright().session( "admin", ( page ) => {
visitRoute( page, "login" )
.fill( "Email", "admin@example.com" )
.fill( "Password", getSystemSetting( "TEST_ADMIN_PASSWORD" ) )
.click( "Sign in" )
} )
}
function run() {
describe( "Users", () => {
it( "shows a user to an admin", () => {
browse( ( page ) => {
visitRoute( page, "users.show", { id : 5 } )
assertRouteIs( page, "users.show" )
expect( page ).toSee( "User 5" )
}, { session : "admin" } )
} )
} )
}
}
| Helper | Does |
|---|---|
routeURL( name, params = {} ) |
Path of a named route (no scheme or host), built by event.route(). Module routes: name@module or module:name. Throws InvalidArgumentException for unknown routes |
visitRoute( page, name, params = {} ) |
page.visit( routeURL( name, params ) ), returns the page. Needs no browser support itself: pages come from browse() |
assertRouteIs( page, name, params = {} ) |
Waits for the page to be on the route. No params: any placeholder value matches, and a route with optional placeholders (/posts/:id?) matches with and without them. With params: the exact path. Ignores case, trailing slash, query and hash. Fails with TestBox.AssertionFailed |
routeURL( "users.show", { id : 5 } ) // /users/5/
routeURL( "home@blog" ) // /blog/home/
assertRouteIs( page, "users.show", { id : 5 } )
Logged-In Tests: Saved Sessions
ColdBox has no test-only login endpoints (no backdoor). Log in once through your real login page with this.playwright().session( name, setup, options ), then start any browse() already logged in with { session : name }:
- The setup page always starts clean; cookies and local storage are saved under the name in the bx-playwright home (
{home}/sessions, outside the project: never commit them). - The session is reused until it is stale:
maxAge(minutes, default0= never expires) orrefresh : truelogs in again. KeepmaxAgebelow the app session timeout. - One session per role (
"admin","editor"); pages of abrowse()without the option start logged out, and every page has its own cookies. - Seed dedicated test users; read passwords from environment variables or CI secrets.
- Test logout like users do: click the link, then
assertRouteIs( page, "login" ).
Plain TestBox (fallback)
Use when @browser support is not available (older TestBox) or in another framework. You manage the manager and artifacts yourself.
describe( "Login", () => {
it( "signs in", () => {
playwright( "ci" ).browse( ( page ) => {
page.visit( "http://localhost:8080/login" )
.fill( "Email", "luis@ortus.com" )
.fill( "Password", "secret" )
.click( "Sign in" )
.assertPathIs( "/dashboard" )
} )
} )
} )
bx-playwright's browse() closes the context with failed = true when the callback throws, so failure artifacts are kept (but not attached to the spec).
Artifacts
Policies for screenshot, trace, video: off, on, only-on-failure, retain-on-failure. Set them with the artifacts setting, a profile (ci, record, debug) or per context:
ctx = pw.newContext( { artifacts : { trace : "retain-on-failure", screenshot : "only-on-failure", video : "retain-on-failure" } } )
page = ctx.newPage()
// ...
kept = ctx.close( failed = true ) // { screenshots : [], trace : "path.zip", videos : [], directory }
View traces: bxplaywright show-trace path/trace.zip. Debug: playwright( "debug" ) or @browserProfile( "debug" ) (headed, slowMo, all artifacts), BX_PLAYWRIGHT_HEADLESS=false, page.snapshot(), bxplaywright codegen <url>.
Page Objects
// pages/LoginPage.bx
class extends="models.PageObject@bxplaywright" {
url = "/login"
elements = { email : "#email", password : "input[type=password]", submit : "button" }
function at() {
page.assertTitle( "Login" ) // `page` is the bound Page
}
function loginAs( required string email, string password = "secret" ) {
page.fill( "@email", email ).fill( "@password", password ).click( "@submit" )
return page.on( new pages.DashboardPage() )
}
}
home = page.visit( new pages.LoginPage() ).loginAs( "luis@ortus.com" ) // visit: navigate + bind + at()
home.assertSee( "Dashboard" ) // page methods work on page objects
page.on( new pages.LoginPage() ) // bind + at() without navigating
home.element( "heading" ).text() // Locator for an alias
elements become @name selectors on the bound page.
Components
// pages/UserMenu.bx
class extends="models.PageComponent@bxplaywright" {
selector = "nav.user"
elements = { logout : "a.logout" }
function name() {
return text( "span" )
}
}
page.within( new pages.UserMenu(), ( menu ) => menu.click( "@logout" ) )
menu = page.component( new pages.UserMenu() )
All actions and assertions work on a component, scoped to its root.
Macros
playwright().macro( "loginAs", ( page, email ) => page.visit( "/login" ).fill( "Email", email ).click( "Sign in" ) )
page.loginAs( "a@b.com" ).assertSee( "Welcome" ) // return nothing to keep chaining
playwright().macro( "shout", ( locator ) => uCase( locator.text() ), "locator" )
playwright().removeMacro( "loginAs" )
Macros are global. Unknown methods throw Playwright.InvalidOption listing registered macros.
Quality Checks
page.assertNoConsoleErrors( [ "favicon.ico" ] ) // ignore by substring
page.assertNoSmoke( [ "/", "/about", "/pricing" ] ) // HTTP < 400 and no JS errors, all reported at once
violations = page.accessibility( { tags : [ "wcag2a", "wcag2aa" ] } ) // axe-core: [ { id, impact, help, helpUrl, nodes } ]
page.assertNoAccessibilityIssues( { impact : "serious", exclude : [ "color-contrast" ] } )
Visual Regression
page.assertScreenshotMatches( "dashboard", { fullPage : true, mask : [ "@clock" ] } )
page.locator( "@chart" ).assertScreenshotMatches( "chart", { maxDiffPixelRatio : 0.01 } )
- First run (or
update : true, orBX_PLAYWRIGHT_UPDATE_SNAPSHOTS=true) writes the baseline totests/snapshots/<name>.png(settingsnapshots.directory). - A mismatch throws
Playwright.AssertionFailed(a spec failure in TestBox) and writes<name>-actual.pngand<name>-diff.png. - Options:
threshold(0.2),maxDiffPixels,maxDiffPixelRatio,mask,directory, screenshot options. Use thescreenshotprofile for stable images.