agentskillexchange Run repeatable model and agent eval suites and inspect scoring traces with Inspect AI
Run benchmark-style eval suites against models or agents, then inspect scored traces instead of relying on ad hoc chats and gut feel.
Prerequisites
Python environment, inspect-ai package, model provider credentials, evaluation datasets or task definitions, optional sandbox dependencies for agent tasks
Installation
Use the upstream install or setup path that matches your environment:
- git clone https://github.com/UKGovernmentBEIS/inspect_ai.git
- pip install -e ".[dev]"
- uv sync --extra dev
- make hooks
Requirements and caveats from upstream:
- If you use VS Code, you should be sure to have installed the recommended extensions (Python, Ruff, and MyPy). Note that you'll be prompted to install these when you open the project in VS Code.
- The web UI lives in a git submodule at src/inspect_ai/_view/ts-mono/. These steps are only needed if you plan to work on the TypeScript/React frontend — Python-only contributors can skip this entirely.
Basic usage or getting-started notes:
Inspect provides many built-in components, including facilities for prompt engineering, tool usage, multi-turn dialog, and model graded evaluations. Extensions to Inspect (e.g. to support new elicitation and scoring t...
Inspect also includes a collection of over 200 pre-built evaluations ready to run on any model (learn more at https://inspect.aisi.org.uk/evals/).
Run linting, formatting, and tests via
Extracted from upstream docs: https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_ai/HEAD/README.md