# Prove whether a prompt or model variant really won before shipping with promptstats

> Run statistically sound comparisons on eval results so prompt and model changes are judged by confidence bounds, not bar-chart vibes.

- Skill: `agentskillexchange/prove-whether-a-prompt-or-model-variant-really-won-before-sh` (Agent Skill)
- Install (CLI): `npx skillmds@latest add agentskillexchange/prove-whether-a-prompt-or-model-variant-really-won-before-sh`
- Raw SKILL.md: https://api.skillmd.com/api/skills/agentskillexchange/prove-whether-a-prompt-or-model-variant-really-won-before-sh/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: agentskillexchange (https://skillmd.com/u/agentskillexchange)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/agentskillexchange/prove-whether-a-prompt-or-model-variant-really-won-before-sh

---


# Prove whether a prompt or model variant really won before shipping with promptstats

Run statistically sound comparisons on eval results so prompt and model changes are judged by confidence bounds, not bar-chart vibes.

## Prerequisites

Python environment, promptstats package, eval result tables or per-input score arrays, prompt or model experiment outputs to compare

## Installation

Use the upstream install or setup path that matches your environment:
- pip install evalstats
- pip install "evalstats[xlsx]"
- pip install "evalstats[all]"
- pip install "evalstats[lmm]"

Requirements and caveats from upstream:
- If you set method="lmm", analyze() switches to a mixed-effects path (score ~ template + (1|input)) with Wald CIs and parametric rank distributions. By default this uses statsmodels (pure Python, no additional setup re...
- ## Python API
- evalstats main use case is as a Python API, which provides a similar entry point, the analyze() function. Simply pass your benchmark data in the correct format, and pass it to analyze to get a battery of results:

Basic usage or getting-started notes:
- What statistics test should I run in X situation?
- as well as example code (which will, obviously, tend to use evalstats, but
- Running estats.analyze() and then estats.print_analysis_summary(analysis) prints a full statistical report to the terminal, including confidence interval line plots, pairwise comparisons between prompt templates, and...

- Source: https://github.com/ianarawjo/promptstats
- Extracted from upstream docs: https://raw.githubusercontent.com/ianarawjo/promptstats/HEAD/README.md

## Documentation

- https://statsforevals.com

## Source

- [Agent Skill Exchange](https://agentskillexchange.com/skills/prove-whether-a-prompt-or-model-variant-really-won-before-shipping-with-promptstats/)