# A/B test agent skills and MCP changes with Caliper

> Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.

- Skill: `agentskillexchange/a-b-test-agent-skills-and-mcp-changes-with-caliper` (Agent Skill)
- Install (CLI): `npx get-skill add agentskillexchange/a-b-test-agent-skills-and-mcp-changes-with-caliper`
- Raw SKILL.md: https://api.skillmd.com/api/skills/agentskillexchange/a-b-test-agent-skills-and-mcp-changes-with-caliper/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: agentskillexchange (https://skillmd.com/u/agentskillexchange)
- Updated: 2026-11-12
- Page: https://skillmd.com/skills/agentskillexchange/a-b-test-agent-skills-and-mcp-changes-with-caliper

---


# A/B test agent skills and MCP changes with Caliper

Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.

## Prerequisites

Python 3.10+, Caliper CLI, target agent runtime such as Claude Code, Codex, Pi, or Hermes, skills or MCP servers under test

## Installation

Install or set up from the source-backed instructions:

Install the CLI with `pipx install caliper-eval` for direct runs, or add the agent skill with `npx skills@latest add edonadei/caliper`; then create an `.eval.yaml`, run `caliper run ... --k 3`, run an ablated control with `--ablate`, and compare results with `caliper compare`.

- Source: https://github.com/edonadei/caliper

## Documentation

- https://github.com/edonadei/caliper

## Source

- [Agent Skill Exchange](https://agentskillexchange.com/skills/ab-test-agent-skills-mcp-changes-with-caliper/)

