A/B test agent skills and MCP changes with Caliper

Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.

agentskillexchange Updated 61 repo stars

File contents

A/B test agent skills and MCP changes with Caliper

Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.

Prerequisites

Python 3.10+, Caliper CLI, target agent runtime such as Claude Code, Codex, Pi, or Hermes, skills or MCP servers under test

Installation

Install or set up from the source-backed instructions:

Install the CLI with pipx install caliper-eval for direct runs, or add the agent skill with npx skills@latest add edonadei/caliper; then create an .eval.yaml, run caliper run ... --k 3, run an ablated control with --ablate, and compare results with caliper compare.

Documentation

Source

agentskillexchange/skills commit ac971007c9

Frequently asked questions

npx get-skill add agentskillexchange/a-b-test-agent-skills-and-mcp-changes-with-caliper