Benchmark browser agents on repeatable Playwright web tasks with Bananalyzer

Benchmark browser agents on repeatable Playwright web tasks with Bananalyzer

Run a repeatable evaluation suite for browser agents against static web task snapshots instead of judging them from demos or one-off tests.

Prerequisites

Python environment, Playwright browser runtime, pytest-based test execution, a custom AgentRunner implementation, example web task snapshots

Installation

Requirements and caveats from upstream:

• individual website. For an agent to best generalize, we require building a diverse dataset of websites across
• In the future we will support more complex evaluation methods and examples that require multiple steps to complete. The

Basic usage or getting-started notes:

• Banana-lyzer is a CLI tool that runs a set of evaluations against a set of example websites.

• The CLI tool will sequentially run examples against a user defined agent by dynamically constructing a pytest test suite

• AgentRunner exposes the example, and a playwright browser context to use.

• Source: https://github.com/reworkd/bananalyzer

• Extracted from upstream docs: https://raw.githubusercontent.com/reworkd/bananalyzer/HEAD/README.md

Documentation

• https://github.com/reworkd/bananalyzer

Source

• Agent Skill Exchange